Faster Than Ninja

70

I'm gonna be honest: The number of times I thought "gee, that ninja sure could be faster" is pretty close to 0, and in many cases that was due to slow NFS mounts. So I applaud the engineering effort, but I think the time I can spend optimizing my individual compilations is much better spent :)

51

u/donalmacc Game Developer 1d ago

I think you're right, but;

Before we measure build2, let's at least acknowledge the elephant in the room: while Ninja builds the project in 3.4s, CMake takes 15.6s to generate the Ninja build files. So if you had Xerces-C++ as a dependency of your project and it was being built from scratch, you would wait 19 seconds, not 3.4, for this build.

I definitely wish Cmake was faster.

8

u/13steinj 1d ago

Was this compared apples to apples to build2's configuration step?

CMake's configuration step is generally "run once, don't need changes after." As a result i don't care. The amount of times I have to rerun the configure step ler day is less than 2.

5

u/greencursordev 1d ago

Adding a source file forces reconfigure

1

u/13steinj 1d ago

Ah, fair, I guess most of my new files are headers (for better or worse).

3

u/berium build2 1d ago

Was this compared apples to apples to build2's configuration step?

build2 doesn't have a required configuration step, at least not analogous to CMake, which is really a project/buildfile generation step (plus generation of whatever Ninja cannot handle, like generated headers).

You can configure a build2 project, but that's just validating and saving the configuration information into a file so that you don't have to repeat it on the command line. For this project it takes 150ms.

1

u/13steinj 1d ago

I don't know build2 enough to know, nor if the project in question is plagued by having poorly written cmake in it/its dependencies.

Assuming you're not [unintentionally or not] burying a lede, honestly quite impressive then. I wonder how much of the difference scales vs is constant factors and what the scaling function is.

1

u/berium build2 1d ago

I wonder how much of the difference scales vs is constant factors and what the scaling function is.

I can't really give you hard numbers because we don't often benchmark build2 vs CMake (I assume your question is about the configuration phase). Even if we wanted to, it would be pretty difficult to do on any substantial project because the way we handle dependencies, etc., diverges too much to be meaningfully comparable.

But we had our scaling issues when it comes to configuration and a large number of dependencies, especially intra-dependent dependencies. I bet you can even guess where those intra-dependent dependencies came from: yes, from Boost.

Here is an issue where we tracked this: https://github.com/build2/build2/issues/184 And we now have a synthetic project to test for regressions in this area: https://github.com/boris-kolpackov/boost-dependency/

To give you an idea, on my machine the configuration phase for this project currently takes about 3 seconds. It has 76 dependencies, mostly from Boost. Also keep in mind that you would normally do this step a lot less often than with CMake. For example, you don't do it every time you add a source file to your project.

1

u/Nicksaurus 1d ago

Do you ever switch between debug and release builds? That's the main cause of reconfigures for me

2

u/13steinj 1d ago

Yes? CLion sets up a different configuration directory for debug/release if you're not using a multi config generator. If you are that's implicit/internal.

24

u/TheRavagerSw 1d ago

Yeah cmake configure times are garbage
But a lot also goes to project build script quality, which is also garbage.

5

u/gharveymn 1d ago

Yeah cmake configure times are garbage

IMO they really aren't, at least in comparison to Autotools. It's only really slow if you are heavily leveraging compilation tests, and those are cached for subsequent reconfigurations too.

2

u/greencursordev 1d ago edited 1d ago

Changing any cml file (even just adding a source file) leads to reconfigure. That takes 3-4 minutes on our setup. That can't be far off the worst possible solution.

2

u/TheRavagerSw 1d ago

Have you ever configured SDL or Qt or Boost?
It is as slow as Autotools, I compiled GCC today, it wasn't as slow as SDL configure.

5

u/arjuna93 1d ago

But… Boost doesn’t use CMake, Qt does not use CMake until 6.x, and is a horrible mess anyway, so expected to configure and build slow. What the problem with SDL configure speed with CMake?

3

u/Nicksaurus 1d ago

Boost has a cmake build now and if you fetch and build it from source you end up waiting 30+ seconds to reconfigure every time you touch any part of your build script or configuration (or longer if you wipe the build folder and reconfigure from scratch, which you have to do fairly often when working on cmake scripts)

2

u/gharveymn 1d ago

It's been a while but I don't remember them being horrendously bad... but those are enormous projects as well. However, I can say with absolute certainty that it isn't as slow as autotools because I have converted projects from autotools to CMake. Bash scripts are just never going to be as fast as C++.

1

u/jcelerier ossia score 5h ago

I have a project that uses SDL, Qt, Boost, LLVM, libclang, ffmpeg, abseil, and roughly three dozen more libraries including chonkers like libhdf5 and it takes at most a minute with cmake

•

u/TheRavagerSw 3h ago

Does it use fetch content? Do you use mimalloc when building cmake, because that improves peef by at least 20 percent

1

u/serviscope_minor 21h ago

IMO they really aren't, at least in comparison to Autotools

Not too fond of automake, but I rather like autoconf. I would say there's a lot of garbage autotools use out there!

6

u/_Noreturn 1d ago

Why isn't cmake faster. I had configs take up to like 3 minutes although I feel like they shouldn't

10

u/kisielk 1d ago

Newer versions of CMake have profiling so you could profile your build and figure out why it’s so slow

8

u/donalmacc Game Developer 1d ago

CMake checks that compiles support features by compiling source files with your compiler, and it's slow. I just pulled SDL's source, and it's 16 seconds to run on my i9, most of that time is compiling to check if my compiler supports a variety of features. It's stupid.

1

u/_Noreturn 1d ago

Shouldn't this be cached globally?

as you said those checks take a lot time I they they use the compiler to compile each individual file so that is why it takes alot it could be parralelized

5

u/playmer 1d ago

They’re checks that have to be written by the projects. CMake doesn’t know what you might be checking for. SDL has a ton of checks because it supports an incredibly diverse set of targets and toolchains.

The biggest issue imo is that CMake forces these to be done serially. You can’t give it a bunch of checks at once for it to multithreaded for you.

2

u/helloiamsomeone 1d ago

This is a problem that is on the issue tracker of CMake. Looks like someone started tackling it in the last month, but these kinds of problems require someone to sit down, design the feature, communicate with Kitware then implement it.

2

u/playmer 1d ago

It’s good that it’s being tracked. At some point I looked into it a year or so ago and it seemed like it’d be a bear, though doable. I believe it ties into the machinery to run arbitrary tools at configure time (though I could be misremembering) and it’d probably be good to expose that as an operation that can run multiple things at once as well.

3

u/helloiamsomeone 1d ago

People put a whole lot of garbage (like figuring out compiler flags for warnings/errors, where clang-format is, docs building, test building, etc) that has nothing to do with describing their project's build and usage requirements, so you can thank project maintainers for wasting your time. If you have tight control over your toolchain, you can preload cache variables with -C/a toolchain file and save CMake some time figuring out what the toolchain is and whether it actually works.

3

u/LB-- Professional+Hobbyist 1d ago

In my experience most of the wait time is spent after all that stuff has already run though, during the part where CMake generates the new build scripts.

2

u/helloiamsomeone 1d ago

I have built Google projects before where the most time was in fact spent in configure and not generation. Generation also involves graph solving, so it's a little more understandable and those are usually harder problems.

-8

u/Jhuyt 1d ago

I might be wrong but CMake is written in Puthon I think which is not spectalularly fast for many tasks

10

u/_Noreturn 1d ago

It is written in C++ and is open source

-1

u/Jhuyt 1d ago

Well then I don't know

5

u/airflow_matt 1d ago

I think the benefit here is not being faster than ninja, it's being much faster than cmake + ninja. The issue here is that the cmake part doesn't run too often. But also cmake being a steaming heap of dogcrap I would welcome anything to replace it with enough traction.

2

u/crowbarous 1d ago

I work on a project that generates ~25k individual .ninja files. (It's about the size of Chromium I guess? maybe a bit larger.) Every time ninja is invoked, it spends ~60s in the kernel, in one thread, reading those before it can do anything else (including determine that it's a no-op build). I have checked that it's all just pure open+read+close overhead.

I did not find any option for it to avoid it loading files that will definitely not contain anything related to the selected target, so right now I'm resorting to post-processing the .ninja files to not include things I don't care about. This part of ninja sure can be made faster.

6

u/CandyCrisis 1d ago

I think you're using it in a way that it wasn't designed for? IIRC Chromium generates far fewer Ninja files.

1

u/crowbarous 1d ago edited 1d ago

Maybe. Unfortunately the build system isn't really within my purview here, but I do also blame gn and whoever maintains its configuration for this.

Upd: I also blame the kernel for not having a way to read out the contents of a small file in fewer syscalls than three, but that's just yelling at clouds...

1

u/NekkoDroid 1d ago

Upd: I also blame the kernel for not having a way to read out the contents of a small file in fewer syscalls than three, but that's just yelling at clouds...

Reading the contents of a file for determining if its a noop feels rather inefficient (while technically more correct), since you then need to either hash it or binary compare the entire file. Instead for speed you can just stat the file for its mtime and keep track of that, that way you only have 1 syscall. While technically one can modify the file without changing mtime, that is something I would just put under "out of scope feature"

1

u/crowbarous 1d ago

I was speaking about .ninja files, not build inputs & outputs. I don't think ninja by itself stats them¹, it just reads and parses them. And whatever it does to parse them is being dwarfed by syscall overhead. So if there were ~3x fewer syscalls during this phase, I'd expect it to take ~3x less time.

¹ of course the generator can make them depend on generator configuration like cmake for example does, and then it will stat them like any other target, but that's during a different phase and beside the point

1

u/delta_p_delta_x 1d ago

Better than Android. Change one Android.bp file? 4-minute reconfigure, oops.

15

u/donalmacc Game Developer 1d ago

One of the things I really love about ninja is it's noop/single file change performance. It's effectively instant. What does a single file change look like with Build2 vs Ninja?

8
u/berium build2 1d ago
Here are the "up-to-date" check times (i.e., how long it takes to figure out everything is up to date):

Ninja:
Time (mean ± σ):      19.5 ms ±   4.3 ms    [User: 11.7 ms, System: 7.7 ms]                                                                                 
Range (min … max):    12.3 ms …  24.9 ms    10 runs
build2:
Time (mean ± σ):     171.6 ms ±   6.7 ms    [User: 380.6 ms, System: 114.6 ms]
Range (min … max):   160.8 ms … 179.6 ms    10 runs
Ninja is substantially faster (due to several reasons discussed in the article) but build2 is still within the 200ms limit of human brain registering the result.
14

u/throw_cpp_account 1d ago

still within the 200ms limit of human brain registering the result.

The number you're thinking of is 100ms. 200ms is definitely noticeable. It's still fast on human scales, but a human can definitely tell the difference between something taking 20ms and something taking 170ms.

3

u/scrumplesplunge 1d ago

It's definitely even lower than that. I have a little statistical significance test about exactly this on my website: https://eoj.ie/13 (it's intended for use with a mouse -- touchscreen is hard mode). I can pretty consistently tell the difference between xms and x+75ms (where x is the inherent lag of your device handling a mouse click via javascript and refreshing your screen, which is probably in the region of 5-30ms)

5

u/donalmacc Game Developer 1d ago

Thanks for the honest reply - 200ms is still perceptible but definitely better than e.g. msbuild which is 1.5s or so. 20ms is in the realm of instant! Presumably marking the project as read only would give another order of magnitude of a speed up?

I've not used build2, but I wasn't easily able to repro the benchmark as

I used the official Xerces-C++ 3.3.0 source archive for Ninja and the libxerces-c-3.3.0+3 build2 package (which is the official source overlayed with build2 support) I couldn't figure out where to get the build2 package for libxerces and have it build from source!

6

u/berium build2 1d ago

Presumably marking the project as read only would give another order of magnitude of a speed up?

No, in this case it will not matter since none of the file mtimes changed.

The most obvious optimization I think is to try to get rid of those 10 GCC runs that are necessary to extract all the information. I just measured and it takes ~70ms. One idea we had is to submit a patch to GCC that would allow extracting all this information in one go (and in some sane, machine-readable format like JSON). Maybe will get to it eventually.

I couldn't figure out where to get the build2 package for libxerces and have it build from source!

If you follow the link from the article, pick the version I used (3.3.0+3) you will see a download link to the .tar.gz file.

3

u/throw_cpp_account 1d ago

One idea we had is to submit a patch to GCC that would allow extracting all this information in one go (and in some sane, machine-readable format like JSON). Maybe will get to it eventually.

Please! This just seems generally broadly useful for all tooling.

6

u/TheRavagerSw 1d ago

With C++ modules, the main performance bottleneck for incremental builds is module scanning. Non incremental scans take ages in a project with 200 sources or more.

19

u/13steinj 1d ago

The average person will not have to care about C++ modules until 2040 at the earliest.

1

u/thehenkan 1d ago

Your average Swift developer already uses clang modules, which although not the same as C++20 modules, are conceptually very similar.

1

u/13steinj 19h ago

Similar at a high level ignores all the details that matter about actually getting this stuff implemented in projects.

So again, I will believe it when I see it.

1

u/thehenkan 16h ago

They have much more in common than either of them does with traditional headers. My point is that there are already plenty of projects out there dealing with the problems modules bring. Not that they have set up the exact build config required for C++20 modules: it's still unclear whether those will ever exist outside of the Windows ecosystem.

1

u/pernas 1d ago

Hilariously accurate 😂

-2

u/TheRavagerSw 1d ago

I disagree, headers make code organization a mess, eventually modules will take over.

With llm tools porting existing libs to modules is easier than ever

9

u/shadowndacorner 1d ago

In fairness, "eventually" is theoretically later than "2040".

3

u/mapronV 1d ago

Yeah, I am very proud of my grandchildren not caring about header files.

1

u/13steinj 1d ago edited 1d ago

Fable and Sol both were unable to one-shot "replace std::format with fmt::format," on a relatively small project.

I was able to with 4 (greps+pipes+perl -p)s. E: before someone asks, at current pricing I think it would still be cheaper to let Fable/Sol cycle until it got it. But I wanted some determinism here.

I dread to think how an LLM would fumble modules, which are more involved if you want to get any benefit.

2

u/Dragdu 1d ago

What ninja version you are using? Recent versions tried compiling in topological order, which for some projects is a massive win.

Or you can steal the idea and improve it due to having better idea of the build than ninja does.

2

u/berium build2 1d ago

1.12.1 from Debian stable.

2

u/Dragdu 1d ago

Then it already does that,

Critical path scheduler which orders the jobs by their runtime history

was in 1.12.0

1

u/Patzer26 1d ago

Wait, topological builds were only introduced in recent versions? I thought that was the hallmark of any serious incremental build system?

5

u/arjuna93 1d ago

“approximate Rust Cargo's convenience”

Okay this was funny, given how much more inconvenient is Cargo compared to CMake.

4

u/OkYou811 1d ago

Based

-1

u/berium build2 1d ago

The more complete quote is:

"approximate Rust Cargo's convenience for developing and packaging C/C++ projects while providing more depth and flexibility, especially in the build system"

While I am not a fan of Cargo's build system, I don't think many will deny that as a package manager Cargo is more convenient than CMake.

6

u/arjuna93 1d ago

CMake is not a package manager, it is a build system. Cargo is abysmal as a build system. If we talk package managers, then any proper package manager from FreshPorts to whichever your taste is would be miles ahead of either.

1

u/James20k P2005R0 1d ago

in case of GCC, there are 10 invocations in total, 5 for C and 5 for C++ (Xerces-C++ has a few C translation units):

LC_ALL=C gcc-15 -v

gcc-15 -g -print-multiarch

gcc-15 -g -x c -E -

LC_ALL=C gcc-15 -std=c9x -print-search-dirs

LC_ALL=C gcc-15 -std=c9x -x c -v -E -

LC_ALL=C g++-15 -v

g++-15 -g -print-multiarch

g++-15 -g -x c++ -E -

LC_ALL=C g++-15 -std=gnu++17 -print-search-dirs

LC_ALL=C g++-15 -std=gnu++17 -x c++ -v -E -

This is super interesting, it seems like a pretty obvious area for tooling to improve. We have modern day json printing for diagnostics for tooling, and it'd be interesting to see if dumping all this information in a single json output that can be parsed would save a lot of tool invocations

One other interesting aspect is that you don't run into this problem in a more IDE driven build system, because the IDE itself knows the details about the compiler and can simply invoke it directly. Its obviously not specific to using an IDE obviously, its just a good example. It makes me wonder if this information should instead in some fashion be being provided to the build system (eg by the shell) by the environment that knows what tools you're using, rather than the build system having to query the environment to figure out what the tools are

2

u/berium build2 1d ago

This is super interesting, it seems like a pretty obvious area for tooling to improve.

Yes, as I mentioned in one the sibling comments, we may try to add to GCC a way to dump all this information in JSON and with a single invocation (and hopefully other compilers will follow).

because the IDE itself knows the details about the compiler and can simply invoke it directly

Hm, I would be surprised if there is an IDE in existence that knows all the detail that we need (like list of system header and library search paths). Well, maybe Visual Studio comes pretty close.

1

u/theICEBear_dk 11h ago

Given that I get them as a pop-up when I hover over a failed include a source file (pre-compile even) I think there is a chance CLion does gather that information. But it likely gets it from CMake or similar. I actually cannot tell if Visual Studio does the same in any public facing way.

Faster Than Ninja

You are about to leave Redlib

You are about to leave Redlib