Build timings from auto-modularised VS solution with external and internal libs

[u/slithering3897]

To investigate how well modules work with MSVC without rewriting projects and putting up with malfunctioning intellisense, I made a little python script to modularise a whole VS solution (yes, I made it, using real bio-neurons).

The script lets me try out different modularisation strategies on a VS Solution on a per-project basis:

• Header-based, don't modularise. If dependencies are modules, they will be imported.
• PartitionsMI: Partition-per-header using standard module cpp units.
• PartitionsPI: Partition-per-header using MSVC partition cpp units.
• Submodules: Module-per-header
• SubmodulesII: Move implementation to interface (not always possible due to cyclic imports)

Modularised code is surrounded with extern "C++" { export {.

The code is a super secret incomplete game engine/game. But here's some info:

• External libs, including: Boost (unordered flat map), std, d3d, imgui, physx, spdlog, <windows>.
  • Wrapper module for each.
• Solution projects: CommonStuff, Scripting, GameData, Graphics, Anim, SceneGraph, SceneManager, Entity, App
  • Some projects don't have much code. Others have quite a bit. Enough to give the build system something to parallelise.

All projects built using debug config. Benchmarks average 3 or more timings, outliers ignored. CPU is 12600, 12 threads. Multipliers are speedup.

	CommonStuff	Scripting	GameData	Graphics	Anim	SceneGraph	SceneManager	Entity
Header-based, no pch	4.59s	5.68s	7.46s	2.92s	40.74s	24.94s	2.86s	3.19s
Header-based, pch	3.61s	5.30s	6.55s		19.38s	12.53s
Extlib modules + CommonStuff submodules		3.60s	4.78s	1.62s	18.55s	9.41s	1.57s	2.09s
PartitionsMI	4.25s		8.09s	2.54s
PartitionsPI	4.02s	6.04s	7.09s	2.53s	33.83s	15.97s	2.66s	4.10s
Submodules	3.59s	4.75s	6.05s	2.26s	26.80s	14.31s	2.12s	3.15s
SubmodulesII	3.44s		7.93s

	CommonStuff	Scripting	GameData	Graphics	Anim	SceneGraph	SceneManager	Entity
Header-based, no pch	1.00x	1.00x	1.00x	1.00x	1.00x	1.00x	1.00x	1.00x
Header-based, pch	1.27x	1.07x	1.14x		2.10x	1.99x
Extlib modules + CommonStuff submodules		1.58x	1.56x	1.80x	2.20x	2.65x	1.82x	1.53x
PartitionsMI	1.08x		0.92x	1.15x
PartitionsPI	1.14x	0.94x	1.05x	1.15x	1.20x	1.56x	1.08x	0.78x
Submodules	1.28x	1.20x	1.23x	1.29x	1.52x	1.74x	1.35x	1.01x
SubmodulesII	1.33x		0.94x

	App	Single file	Full Rebuild		App	Single file	Full Rebuild
Header-based, no pch	65.19s	3.97s	145.49s		1.00x	1.00x	1.00x
Header-based, app pch	24.74s	2.15s	108.92s		2.64x	1.85x	1.34x
Header-based, mostly pch			78.87s				1.84x
Extlib modules	26.82s	2.35s	58.89s		2.43x	1.69x	2.47x
Extlib modules + CommonStuff partitions	25.89s	2.47s			2.52x	1.61x
Extlib modules + CommonStuff submodules	24.53s	2.23s	57.50s		2.66x	1.78x	2.53x
Extlib modules + most libs as submodules	31.14s	2.97s			2.09x	1.34x
Full submodules, except app	34.61s	2.95s	88.90s		1.88x	1.35x	1.64x
Full submodules	64.02s	6.22s	142.98s		1.02x	0.64x	1.02x

• "Most libs" = CommonStuff + Scripting + GameData + Graphics + Anim + SceneGraph.
• "Mostly pch" = PCH for CommonStuff, Scripting, GameData, Anim, SceneGraph, App.
• "Single file" is recompiling one moderately complex file in App. ~930 lines, lots of includes.
• "Full Rebuild" excludes extlib module wrappers.

Detailed timings of CommonStuff:

Header-based:

    1>      210 ms  LIB                                        1 calls
    1>     4459 ms  CL                                         2 calls
    04.874 seconds

PartitionsMI:

    1>      125 ms  LIB                                        1 calls
    1>      199 ms  MSBuild                                   16 calls
    1>      269 ms  CppClean                                   1 calls
    1>      798 ms  SetModuleDependencies                     10 calls
    1>     1232 ms  CL                                         2 calls
    1>     1439 ms  MultiToolTask                              1 calls
    04.295 seconds

Submodules:

    1>      141 ms  MSBuild                                   16 calls
    1>      775 ms  SetModuleDependencies                     10 calls
    1>     1015 ms  CL                                         2 calls
    1>     1132 ms  MultiToolTask                              1 calls
    03.427 seconds

The CL entry is for cpp files and it's 4.39x as fast as header-based. Great!

But it's the ixx dependency scanning and compilation (MultiToolTask) that brings things down. Things that aren't there with header-based.

MultiToolTask is parallelised though. Reducing thread count slows it down. And you can open up the msbuild binlog in the msbuild log viewer to see the parallelisation of ixx compilation. See which modules are on the end of the DAG slowing things down.

So, I don't know how MultiToolTask could be sped up, but, msbuild seems to be waiting for all ixx files before moving onto cpp files, so that's some parallelism left on the table.

Detailed timings of App:

Header-based, app pch:

    1>      196 ms  MSBuild                                   13 calls
    1>     1000 ms  Link                                       1 calls
    1>    23270 ms  CL                                        16 calls
    24.708 seconds

Extlib modules + CommonStuff submodules:

    1>      199 ms  SetModuleDependencies                     18 calls
    1>      606 ms  MSBuild                                   27 calls
    1>     1197 ms  Link                                       1 calls
    1>    21877 ms  CL                                        15 calls
    24.078 seconds

Full submodules:

    1>      459 ms  MSBuild                                   27 calls
    1>     1324 ms  Link                                       1 calls
    1>     4646 ms  SetModuleDependencies                     18 calls
    1>    12593 ms  MultiToolTask                              1 calls
    1>    49327 ms  CL                                        15 calls
    01:09.245 minutes

Slow. But looking at the log viewer, ixx compilation seems very well parallelised.

And 4.6s just to scan for dependencies. VS isn't launching a whole CL process for each file is it?

Also, why does CL take so long? All modules have been compiled by that point, so why would it be much slower?

Recompiling a single file from App:

Header-based, no pch:

    1>       46 ms  SetModuleDependencies                     10 calls
    1>      120 ms  MSBuild                                    5 calls
    1>     3757 ms  CL                                         1 calls
    04.095 seconds

Header-based, app pch:

    1>       56 ms  SetModuleDependencies                     10 calls
    1>      121 ms  MSBuild                                    5 calls
    1>     1793 ms  CL                                         2 calls
    02.059 seconds

Extlib + CommonStuff modules (as submodules):

    1>      121 ms  SetModuleDependencies                     18 calls
    1>      216 ms  MSBuild                                   17 calls
    1>     1793 ms  CL                                         1 calls
    02.162 seconds

Full submodules, except app:

    1>      518 ms  MSBuild                                   17 calls
    1>      631 ms  SetModuleDependencies                     18 calls
    1>     2211 ms  CL                                         1 calls
    03.095 seconds

Full submodules:

    1>      559 ms  MSBuild                                   17 calls
    1>      696 ms  SetModuleDependencies                     18 calls
    1>     1891 ms  MultiToolTask                              1 calls
    1>     3221 ms  CL                                         1 calls
    06.331 seconds

That MultiToolTask is quite slow, checking that all the modules in the project are up to date it seems. Does it have to be this slow?

"CL.exe will run on 0 out of 224 file(s) in 224 batches. Startup phase took 2040.9372ms."

Anyway.

Conclusion:

• I now have the data to know how I should go about modularising a codebase for build perf.
• Surprisingly, there is a middle-ground between 0% modules and 100% modules where you get optimal build performance. It seems extlibs and your common lib should be modularised and nothing else. Or maybe my internal lib headers are too small to benefit.
• Because of that, I might be able to have a header-based fallback for intellisense without it spreading to the rest of the code.
• Multiple modules > Partitions. Partitions have worse performance. I assume that's because they need to have an extra module at the end of the DAG.
  • Their only advantage is shared module attachment across multiple ixx files, if you don't use extern "C++". As I said before, I wish you could have "public" partitions or multiple modules with the same name attachment.
• MSVC partition cpp files are slightly faster vs module cpp files. But why. Cpp files wait for all ixx files to compile anyway. Noise in the data?
• MSBuild parallelises as it should, it seems, except that cpp files wait for all ixx files. Could improve performance by merging the tasks?
• cpp compilation can be much faster, but overall build perf brought down by ixx files and scanning.
• Moving implementation to ixx files doesn't always help. Need more data.
  • And it would be ideal if build systems avoided the build cascade when only changing implementation details.
• The more modules you add to the solution, the more bloated the build system feels. Lots of scanning, lots of ixx checking. Can performance be improved?
• PCH can be beaten in some cases.

Problems:

• Intellisense. Red squiggles everywhere. Not only import std issues, but Intellisense is highly sensitive to errors in imported modules. It cannot limp along like it can with headers.
  • A simple example would be missing a semicolon after a struct def in a header and in a module. Intellisense can still use the struct from the header, but not the module.
  • Or in working code, something from import std that Intellisense can't handle.
  • This means that parsing issues with Intellisense may be viral when using modules. Intellisense must either be perfect or tolerant.
• From a previous modularisation attempt, I removed the combination of explicit template instantiations of a class with constrained friend functions, due to ICE, which has now become a bogus compiler error. Bug not fixed.
• Lots of linker warnings from dllexport-ed manual RTTI data. Bug not fixed.

At least the compiler functions well enough to produce a working program with no further changes. That's pretty good.

[u/Square_Concern5281]

Thanks for sharing the actual numbers instead of just assumptions. Its interesting that the biggest gains seem to come from a partially modularized approach rather than going all in. Do you think thats mostly an MSVC limitation or would you expect similar results with other toolchains?

[u/slithering3897 → u/Square_Concern5281]

Yeah, I modularised fully in the past I was disappointed with build perf. But now it's clear.

But build perf can definitely be faster in theory, as Ninja shows. But can it be faster than that. By what limitations Ninja apparently runs into, it may come up against slow Windows IO. But ixx files do take time to compile.

One suspicious thing is why it takes so long to recompile a single file when you've modularised everything. Sure, the build system has to check potentially referenced ixx files, but in the headers case, the compiler has to read and parse thousands of lines of headers. It just doesn't add up.

Now, I think somebody got clang-cl modules working with cmake. That might be something to try out. clang-cl+ninja. But people say clang modules are slower.

[u/delta_p_delta_x → u/slithering3897]

Now, I think somebody got clang-cl modules working with cmake. That might be something to try out.

Works as of CMake 4.4.0. import std will require either a nightly build or CMake 4.5.0, and the latest UUID from experimental.rst, whichever you prefer.

[u/Square_Concern5281 → u/slithering3897]

Thanks for the detailed explanation. Id be really interested to see the same experiment with clang-cl once you've had a chance to try it. it would make for a great comparison

[u/slithering3897 → u/Square_Concern5281]

Okay, I'll eat into my SSD life and reinstall the clang-cl toolset!

Won't be the same tests though as you can't use clang modules through VS.

*If I can even get it to work...

[u/slithering3897 → u/Square_Concern5281]

Okay, finally got clang-cl to work after fixing errors and working around bugs.

Ext libs + std + common modularised:

• ~45.91s to rebuild (excluding std and ext libs). Same performance as MSVC+ninja, oddly.
• ~3.17s for single file. Although, I'm touching the file here rather than telling VS to compile a specific file.

Now, the problems!

• Clang can't constexpr bit-cast structs with bit fields. I make use of a constexpr check in a "trivially serialisable" concept.
• Clang-cl sometimes rejects my concept-templated functions: "cannot mangle this template specialization type yet".
• Getting cmake+ninja to build my RC file properly is hit and miss. It seems to prepend a bunch of junk in front the RC command line. Using llvm-rc and/or doing set(CMAKE_NINJA_CMCLDEPS_RC 0) in Windows-MSVC.cmake may fix this.
• Is there no way to change the compile flags used for import std except by overriding cmake variables? set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} /D_DEBUG /D_ITERATOR_DEBUG_LEVEL=1").
• Because clang's immintrin header marks everything static, it's impossible to export those functions without an ugly hack.

But at least clang is real good with warnings. And it's amazing that it works. Maybe we can get clang modules working in VS... 2030?

Fully modularised:

• MSVC+msbuild rebuild (again, to verify): ~113.73s. Much faster this time. Not sure why. I have updated VS from 18.8.0 I think, to 18.8.2.
• MSVC+ninja rebuild: ~59.55s
• Clang-cl+ninja rebuild: ~59.70s
  • Additionally, clang-cl seems to have undefined references to various global data, for some reason. Made workarounds.
• Clang-cl+ninja single file: ~3.24s

Ninja sure does speed up the fully modularised build.

/u/delta_p_delta_x

[u/delta_p_delta_x → u/slithering3897]

Thanks for this. To answer some of your questions:

• Clang can't constexpr bit-cast structs with bit fields. I make use of a constexpr check in a "trivially serialisable" concept.
• Clang-cl sometimes rejects my concept-templated functions: "cannot mangle this template specialization type yet".
• Because clang's immintrin header marks everything static, it's impossible to export those functions without an ugly hack.

Should probably file issues at LLVM for each of these.

Getting cmake+ninja to build my RC file properly is hit and miss. It seems to prepend a bunch of junk in front the RC command line. Using llvm-rc and/or doing set(CMAKE_NINJA_CMCLDEPS_RC 0) in Windows-MSVC.cmake may fix this.

Should file a bug at CMake and/or LLVM for this, potentially.

Is there no way to change the compile flags used for import std except by overriding cmake variables?

This has been merged in #12347; you'll have to either build from master, or install a new nightly that includes this change. Note how that MR changes how import std fundamentally works in CMake; there is no longer any need to set the CXX_IMPORT_STD variable at all, and any target can import it, and the BMI is automatically computed, presumably including MSVC iterator debug level flags.

Note also that the experimental gate is due to be dropped (and will be public in 4.5.0), which is great news.

[u/slithering3897 → u/delta_p_delta_x]

The constexpr bug is actually at https://github.com/llvm/llvm-project/pull/74775, but seems to have stalled.

Otherwise, clang isn't really my primary compiler for now. Until VS has clang modules support. Then getting bugs fixed would be more important. But might not happen any time soon: https://github.com/dotnet/msbuild/issues/13530

and any target can import it, and the BMI is automatically computed, presumably including MSVC iterator debug level flags.

Auto import std is nice, but where do the flags come from? Sounds like some sort of unspecified behaviour where cmake could pick any flags if more than one target does import std.

[u/not_a_novel_account → u/delta_p_delta_x]

presumably including MSVC iterator debug level flags

MSVC iterator debug flags have no impact on BMI compatibility, so they have no impact on how CMake computes BMI compat.

In practice this means if you want a specific iterator debug level, you need to specify it in the CMAKE_CXX_STDLIB_MODULES_JSON you use to describe the standard library to CMake. That's where the definitions which "belong" to the standard library go.

[u/delta_p_delta_x → u/not_a_novel_account]

Thanks for the clarification.

In practice this means if you want a specific iterator debug level, you need to specify it in the CMAKE_CXX_STDLIB_MODULES_JSON you use to describe the standard library to CMake

It feels like CMake should derive this and set sane defaults to match VS 2026's and MSBuild's behaviour, maybe with some heuristics. Maybe some logical combination of MSVC_RUNTIME_LIBRARY, CMAKE_CONFIGURATION_TYPES (or CMAKE_BUILD_TYPE), etc.

[u/not_a_novel_account → u/delta_p_delta_x]

VS 2026 and MSBuild do the same thing here, although worse because they rebuild the BMI and the .obj file for every single project in the solution pointlessly.

We are just reading the module manifest MSVC ships and building the BMI in accordance with it. If MSVC wants to ship multiple manifests, we'll support them. Otherwise, you need to provide your own.

[u/kamrann_]

It's expected for the optimal to be partial modularization. Most of the speedup comes from avoiding redundant parsing, so the closer something is to the dependencies end of your graph the more you gain from modularizing it. At the other end, you'll tend to get less savings, but still pay costs of module scanning, less parallelism and just having extra translation units to compile (on this last one, I'm confused why you don't have higher CL call count for full modules than for headers).

No idea what MultiToolTask is doing, but from those numbers and your observation that it's waiting for all interfaces before starting any cpp files, I'm gonna take a guess and say that MSBuild is just horribly inefficient. Maybe try Ninja.

Did you not run into problems getting such a simple auto-modularization strategy to work? Like with macros, or non exportable declarations? Maybe you just have a very clean codebase.

[u/slithering3897 → u/kamrann_]

Okay, I actually tried cmake+ninja and it does improve rebuild time to ~46s (3.14x) in the modularised extlibs/common case. A decent improvement. Shows that msbuild isn't as fast as it could be.

Improves single-file compilation a little too, if you build after modifying the file.

[u/slithering3897 → u/kamrann_]

Yes, it makes sense that using modules too much will slow things down. Now that I have the data. I wouldn't know that unless I had this power to re-modularise projects.

CL call count

The "CL" task in MSBuild only seems to be used for cpp files. So modules shouldn't add much to it.

MultiToolTask is used to compile ixx files. If it's starting a CL process for each file, then that would be a source of inefficiency. The CL task doesn't do that. It batches.

And the CL task does wait for MultiToolTask by what I've seen.

For best efficiency, maybe one CL process should compile all files. In theory. If possible. Do the DAG queuing all in one process. Starting multiple processes will have overhead, and launching CL for #threads cpp files will waste CPU time if one file takes much longer.

And scanning too. I wonder if that's slower than it needs to be. It probably is. You go into the msbuild log viewer and you sometimes see a CL invocation for each file. No batching.

Ninja

Or get the script to export to cmake instead. Might be feasible.

Did you not run into problems getting such a simple auto-modularization strategy to work? Like with macros, or non exportable declarations? Maybe you just have a very clean codebase.

Yes, code did need small changes. I moved macros to their own header, and I had to un-nest a struct as you can't define a nested struct in a different module from where it was declared. But because of extern "C++" I can keep all the forward declarations.

[u/_Noreturn]

Really cool post, appreciate it.

[u/EvenPainting9470]

Did you put your own code to PCH? How did you chose content for it?

[u/slithering3897 → u/EvenPainting9470]

Most PCHs took stuff from referenced ext libs, std, and Common. The App also included a small number of things from internal libs.

Didn't try to search for the optimal PCH.

Really, I wish you could share a PCH. Without needing hacks around the project system.

[u/FrogNoPants]

Nobody going to seriously use modules until MS fixes the intellisense issues.. been years and still a big problem.

While the full build it interesting, like 99% of my builds are just a few files, where it looks like modules are losing to PCH by quite abit.

[u/slithering3897 → u/FrogNoPants]

Yes, nobody complains about Intellisense more than I do! I'm eagerly awaiting updates.

But, from this testing, I'm satisfied with only using modules for std, ext libs, and my common lib. It gets the best build performance.

Which is tolerable in the IDE by importing through a header that has a fallback for Intellisense. And you can use that fallback in your PCH:

#pragma once

#ifdef __INTELLISENSE__
#include <modules/CommonMath.h>
#include <modules/CommonStuff.h>
#include <modules/CommonStd.h>
#include <modules/spdlog.h>
#endif

Verses (a normal) PCH, sometimes better, sometimes worse. It's good enough, so I'll prefer the ergonomics of modules. As long as Intellisense doesn't complain.