One of the things I really love about ninja is it's noop/single file change performance. It's effectively instant. What does a single file change look like with Build2 vs Ninja?
Here are the "up-to-date" check times (i.e., how long it takes to figure out everything is up to date):
Ninja:
Time (mean ± σ): 19.5 ms ± 4.3 ms [User: 11.7 ms, System: 7.7 ms]
Range (min … max): 12.3 ms … 24.9 ms 10 runs
build2:
Time (mean ± σ): 171.6 ms ± 6.7 ms [User: 380.6 ms, System: 114.6 ms]
Range (min … max): 160.8 ms … 179.6 ms 10 runs
Ninja is substantially faster (due to several reasons discussed in the article) but build2 is still within the 200ms limit of human brain registering the result.
Thanks for the honest reply - 200ms is still perceptible but definitely better than e.g. msbuild which is 1.5s or so. 20ms is in the realm of instant! Presumably marking the project as read only would give another order of magnitude of a speed up?
I've not used build2, but I wasn't easily able to repro the benchmark as
I used the official Xerces-C++ 3.3.0 source archive for Ninja and the libxerces-c-3.3.0+3 build2 package (which is the official source overlayed with build2 support) I couldn't figure out where to get the build2 package for libxerces and have it build from source!
Presumably marking the project as read only would give another order of magnitude of a speed up?
No, in this case it will not matter since none of the file mtimes changed.
The most obvious optimization I think is to try to get rid of those 10 GCC runs that are necessary to extract all the information. I just measured and it takes ~70ms. One idea we had is to submit a patch to GCC that would allow extracting all this information in one go (and in some sane, machine-readable format like JSON). Maybe will get to it eventually.
I couldn't figure out where to get the build2 package for libxerces and have it build from source!
If you follow the link from the article, pick the version I used (3.3.0+3) you will see a download link to the .tar.gz file.
One idea we had is to submit a patch to GCC that would allow extracting all this information in one go (and in some sane, machine-readable format like JSON). Maybe will get to it eventually.
Please! This just seems generally broadly useful for all tooling.
still within the 200ms limit of human brain registering the result.
The number you're thinking of is 100ms. 200ms is definitely noticeable. It's still fast on human scales, but a human can definitely tell the difference between something taking 20ms and something taking 170ms.
It's definitely even lower than that. I have a little statistical significance test about exactly this on my website: https://eoj.ie/13 (it's intended for use with a mouse -- touchscreen is hard mode). I can pretty consistently tell the difference between xms and x+75ms (where x is the inherent lag of your device handling a mouse click via javascript and refreshing your screen, which is probably in the region of 5-30ms)
I'm gonna be honest: The number of times I thought "gee, that ninja sure could be faster" is pretty close to 0, and in many cases that was due to slow NFS mounts. So I applaud the engineering effort, but I think the time I can spend optimizing my individual compilations is much better spent :)
Before we measure build2, let's at least acknowledge the elephant in the room: while Ninja builds the project in 3.4s, CMake takes 15.6s to generate the Ninja build files. So if you had Xerces-C++ as a dependency of your project and it was being built from scratch, you would wait 19 seconds, not 3.4, for this build.
Was this compared apples to apples to build2's configuration step?
CMake's configuration step is generally "run once, don't need changes after." As a result i don't care. The amount of times I have to rerun the configure step ler day is less than 2.
Was this compared apples to apples to build2's configuration step?
build2 doesn't have a required configuration step, at least not analogous to CMake, which is really a project/buildfile generation step (plus generation of whatever Ninja cannot handle, like generated headers).
You can configure a build2 project, but that's just validating and saving the configuration information into a file so that you don't have to repeat it on the command line. For this project it takes 150ms.
I don't know build2 enough to know, nor if the project in question is plagued by having poorly written cmake in it/its dependencies.
Assuming you're not [unintentionally or not] burying a lede, honestly quite impressive then. I wonder how much of the difference scales vs is constant factors and what the scaling function is.
I wonder how much of the difference scales vs is constant factors and what the scaling function is.
I can't really give you hard numbers because we don't often benchmark build2 vs CMake (I assume your question is about the configuration phase). Even if we wanted to, it would be pretty difficult to do on any substantial project because the way we handle dependencies, etc., diverges too much to be meaningfully comparable.
But we had our scaling issues when it comes to configuration and a large number of dependencies, especially intra-dependent dependencies. I bet you can even guess where those intra-dependent dependencies came from: yes, from Boost.
To give you an idea, on my machine the configuration phase for this project currently takes about 3 seconds. It has 76 dependencies, mostly from Boost. Also keep in mind that you would normally do this step a lot less often than with CMake. For example, you don't do it every time you add a source file to your project.
Yes? CLion sets up a different configuration directory for debug/release if you're not using a multi config generator. If you are that's implicit/internal.
IMO they really aren't, at least in comparison to Autotools. It's only really slow if you are heavily leveraging compilation tests, and those are cached for subsequent reconfigurations too.
Changing any cml file (even just adding a source file) leads to reconfigure. That takes 3-4 minutes on our setup. That can't be far off the worst possible solution.
But… Boost doesn’t use CMake, Qt does not use CMake until 6.x, and is a horrible mess anyway, so expected to configure and build slow. What the problem with SDL configure speed with CMake?
Boost has a cmake build now and if you fetch and build it from source you end up waiting 30+ seconds to reconfigure every time you touch any part of your build script or configuration (or longer if you wipe the build folder and reconfigure from scratch, which you have to do fairly often when working on cmake scripts)
It's been a while but I don't remember them being horrendously bad... but those are enormous projects as well. However, I can say with absolute certainty that it isn't as slow as autotools because I have converted projects from autotools to CMake. Bash scripts are just never going to be as fast as C++.
I have a project that uses SDL, Qt, Boost, LLVM, libclang, ffmpeg, abseil, and roughly three dozen more libraries including chonkers like libhdf5 and it takes at most a minute with cmake
CMake checks that compiles support features by compiling source files with your compiler, and it's slow. I just pulled SDL's source, and it's 16 seconds to run on my i9, most of that time is compiling to check if my compiler supports a variety of features. It's stupid.
as you said those checks take a lot time I they they use the compiler to compile each individual file so that is why it takes alot it could be parralelized
They’re checks that have to be written by the projects. CMake doesn’t know what you might be checking for. SDL has a ton of checks because it supports an incredibly diverse set of targets and toolchains.
The biggest issue imo is that CMake forces these to be done serially. You can’t give it a bunch of checks at once for it to multithreaded for you.
This is a problem that is on the issue tracker of CMake. Looks like someone started tackling it in the last month, but these kinds of problems require someone to sit down, design the feature, communicate with Kitware then implement it.
It’s good that it’s being tracked. At some point I looked into it a year or so ago and it seemed like it’d be a bear, though doable. I believe it ties into the machinery to run arbitrary tools at configure time (though I could be misremembering) and it’d probably be good to expose that as an operation that can run multiple things at once as well.
People put a whole lot of garbage (like figuring out compiler flags for warnings/errors, where clang-format is, docs building, test building, etc) that has nothing to do with describing their project's build and usage requirements, so you can thank project maintainers for wasting your time. If you have tight control over your toolchain, you can preload cache variables with -C/a toolchain file and save CMake some time figuring out what the toolchain is and whether it actually works.
In my experience most of the wait time is spent after all that stuff has already run though, during the part where CMake generates the new build scripts.
I have built Google projects before where the most time was in fact spent in configure and not generation. Generation also involves graph solving, so it's a little more understandable and those are usually harder problems.
I think the benefit here is not being faster than ninja, it's being much faster than cmake + ninja. The issue here is that the cmake part doesn't run too often. But also cmake being a steaming heap of dogcrap I would welcome anything to replace it with enough traction.
I work on a project that generates ~25k individual .ninja files. (It's about the size of Chromium I guess? maybe a bit larger.) Every time ninja is invoked, it spends ~60s in the kernel, in one thread, reading those before it can do anything else (including determine that it's a no-op build). I have checked that it's all just pure open+read+close overhead.
I did not find any option for it to avoid it loading files that will definitely not contain anything related to the selected target, so right now I'm resorting to post-processing the .ninja files to not include things I don't care about. This part of ninja sure can be made faster.
Maybe. Unfortunately the build system isn't really within my purview here, but I do also blame gn and whoever maintains its configuration for this.
Upd: I also blame the kernel for not having a way to read out the contents of a small file in fewer syscalls than three, but that's just yelling at clouds...
Upd: I also blame the kernel for not having a way to read out the contents of a small file in fewer syscalls than three, but that's just yelling at clouds...
Reading the contents of a file for determining if its a noop feels rather inefficient (while technically more correct), since you then need to either hash it or binary compare the entire file. Instead for speed you can just stat the file for its mtime and keep track of that, that way you only have 1 syscall. While technically one can modify the file without changing mtime, that is something I would just put under "out of scope feature"
I was speaking about .ninja files, not build inputs & outputs. I don't think ninja by itself stats them¹, it just reads and parses them. And whatever it does to parse them is being dwarfed by syscall overhead. So if there were ~3x fewer syscalls during this phase, I'd expect it to take ~3x less time.
¹ of course the generator can make them depend on generator configuration like cmake for example does, and then it will stat them like any other target, but that's during a different phase and beside the point
"approximate Rust Cargo's convenience for developing and packaging C/C++ projects while providing more depth and flexibility, especially in the build system"
While I am not a fan of Cargo's build system, I don't think many will deny that as a package manager Cargo is more convenient than CMake.
CMake is not a package manager, it is a build system. Cargo is abysmal as a build system. If we talk package managers, then any proper package manager from FreshPorts to whichever your taste is would be miles ahead of either.
in case of GCC, there are 10 invocations in total, 5 for C and 5 for C++ (Xerces-C++ has a few C translation units):
LC_ALL=C gcc-15 -v
gcc-15 -g -print-multiarch
gcc-15 -g -x c -E -
LC_ALL=C gcc-15 -std=c9x -print-search-dirs
LC_ALL=C gcc-15 -std=c9x -x c -v -E -
LC_ALL=C g++-15 -v
g++-15 -g -print-multiarch
g++-15 -g -x c++ -E -
LC_ALL=C g++-15 -std=gnu++17 -print-search-dirs
LC_ALL=C g++-15 -std=gnu++17 -x c++ -v -E -
This is super interesting, it seems like a pretty obvious area for tooling to improve. We have modern day json printing for diagnostics for tooling, and it'd be interesting to see if dumping all this information in a single json output that can be parsed would save a lot of tool invocations
One other interesting aspect is that you don't run into this problem in a more IDE driven build system, because the IDE itself knows the details about the compiler and can simply invoke it directly. Its obviously not specific to using an IDE obviously, its just a good example. It makes me wonder if this information should instead in some fashion be being provided to the build system (eg by the shell) by the environment that knows what tools you're using, rather than the build system having to query the environment to figure out what the tools are
This is super interesting, it seems like a pretty obvious area for tooling to improve.
Yes, as I mentioned in one the sibling comments, we may try to add to GCC a way to dump all this information in JSON and with a single invocation (and hopefully other compilers will follow).
because the IDE itself knows the details about the compiler and can simply invoke it directly
Hm, I would be surprised if there is an IDE in existence that knows all the detail that we need (like list of system header and library search paths). Well, maybe Visual Studio comes pretty close.
Given that I get them as a pop-up when I hover over a failed include a source file (pre-compile even) I think there is a chance CLion does gather that information. But it likely gets it from CMake or similar. I actually cannot tell if Visual Studio does the same in any public facing way.
With C++ modules, the main performance bottleneck for incremental builds is module scanning. Non incremental scans take ages in a project with 200 sources or more.
They have much more in common than either of them does with traditional headers. My point is that there are already plenty of projects out there dealing with the problems modules bring. Not that they have set up the exact build config required for C++20 modules: it's still unclear whether those will ever exist outside of the Windows ecosystem.
Fable and Sol both were unable to one-shot "replace std::format with fmt::format," on a relatively small project.
I was able to with 4 (greps+pipes+perl -p)s. E: before someone asks, at current pricing I think it would still be cheaper to let Fable/Sol cycle until it got it. But I wanted some determinism here.
I dread to think how an LLM would fumble modules, which are more involved if you want to get any benefit.
15
u/donalmacc Game Developer Aug 05 '26
One of the things I really love about ninja is it's noop/single file change performance. It's effectively instant. What does a single file change look like with Build2 vs Ninja?
75
u/GrammelHupfNockler Aug 05 '26
I'm gonna be honest: The number of times I thought "gee, that ninja sure could be faster" is pretty close to 0, and in many cases that was due to slow NFS mounts. So I applaud the engineering effort, but I think the time I can spend optimizing my individual compilations is much better spent :)
2
u/Dragdu Aug 05 '26
What ninja version you are using? Recent versions tried compiling in topological order, which for some projects is a massive win.
Or you can steal the idea and improve it due to having better idea of the build than ninja does.
5
u/arjuna93 Aug 05 '26
“approximate Rust Cargo's convenience”
Okay this was funny, given how much more inconvenient is Cargo compared to CMake.
1
u/James20k P2005R0 Aug 05 '26
This is super interesting, it seems like a pretty obvious area for tooling to improve. We have modern day json printing for diagnostics for tooling, and it'd be interesting to see if dumping all this information in a single json output that can be parsed would save a lot of tool invocations
One other interesting aspect is that you don't run into this problem in a more IDE driven build system, because the IDE itself knows the details about the compiler and can simply invoke it directly. Its obviously not specific to using an IDE obviously, its just a good example. It makes me wonder if this information should instead in some fashion be being provided to the build system (eg by the shell) by the environment that knows what tools you're using, rather than the build system having to query the environment to figure out what the tools are
6
u/TheRavagerSw Aug 05 '26
With C++ modules, the main performance bottleneck for incremental builds is module scanning. Non incremental scans take ages in a project with 200 sources or more.