[Ninja author here] Nice post, cool to see the deep dive! I also appreciate the details on how they produced their numbers.
As they observe, Ninja gets to be fast mostly by cheating: it avoids a lot of work by saying many things are just out of scope for Ninja to do, and that means it is a useful a target to race against. (Funny thing: when I wrote Ninja I was misremembering how fast an earlier build system was so I kept trying to make it faster. So don't treat it as a lower bound, I just made it up!)
I comment here to say I find the explanation for 'why' in this post unsatisfying. They mention three design decisions.
The first one is a criticism of CMake, not Ninja (?), so I don't think it can be why. I might have misunderstood?
The second reason given is doing some work like header dependencies in multiple threads. This is the most plausible reason to me but it still feels unlikely. It's a very small amount of work: the post mentions 300 compiles, so maybe parsing 300 small text files?
The third is that they run the compiler up front an additional time to gather headers, which is strictly more work than Ninja. There is some hand waving about file access patterns but I am skeptical; if the end-to-end build time is 3 seconds then the project is small enough to all fit in kernel caches. They also mention doing other things like invoking the compiler to get version information. This seems like it would dwarf any performance gain from number 2.
Maybe it's just my own curiosity, I think this post would be better if it had a better explanation for the reason. I'm not disputing the result, I just think the result should make you suspicious that something else is going on, and you might learn something from that! You could for example explore whether it's the header dependency thing by profiling the Ninja invocation and seeing if it's waiting for CPU or waiting for tasks to execute.
(If I had to guess without looking at any of the involved code, I would predict it's something about how CMake generates the build, like it introduces serialization in a place where build2 is parallel, or it adds some extra build steps like gathering the current git hash into a header file or something.)
Simple CMake projects using the Ninja generator are very efficient, unless you create generated files.
And if they are in their own targets, they are not really an issue (they would serialize everything that depends on them as you'd expect), but if you have them in a library grouped with other files to compile, then the whole library compilation is serialized.
And obviously worse if you also have to build the generator for the generated files, but that's not a big surprise, you can observe that in full builds of Chromium or its libraries too waiting for protoc if you crank the parallelization a lot.
Yes, I don’t remember the details but vaguely remember that CMake tends to group things together that could in principle be made more parallel, as you mention with generates files in a library. On the other hand if build2 makes it easier for authors to express these kinds of patterns without the serialization then I count that as a win for build2!
Always glad to see authors on here. Thanks for your insight!
Since they had to rewrite the build file for their program I also assume that something is missing. Didn’t see any mention of verifying that.
I also really didn’t like their denigrating tone. It totally turns me off trying build2, because it seems they don’t understand the point of separating build stages like environment setup (getting dependencies), configure, native build, cross build, packaging. I am a very happy ninja user instead of a batteries-included solution because it does its one job well and can be used very flexibly. I personally detest cmake, so I use nix + own configure script + ninja.
The blog is also wrong about ninja being unable to call configure, but I intentionally don’t want that (I want the build stages to communicate in one direction for sanity).
> I want the build stages to communicate in one direction for sanity
That's easy to do if you control the whole pipeline and can integrate all the features together, not so much with the CMake model unfortunately. I think it would be nice if CMake had Ninja integrated as a library, it could lead to some nice optimizations later.
I wish that when people posted performance results, they would include a theory explaining them. Unexpected performance wins are often noise, bugs, or harness issues. No program is faster just because of the superior manly virtue of its programmer.
Every effect has a cause, and when it comes to performance, it's crucial you understand that cause. If (as in this case) something, from first principles, shouldn't be faster, yet it is, you dig and understand, not just say your shit rules and the other thing sux0rz.
> Let's see if we can go even faster. Next, we disable compression in the file cache. We will discuss the file cache in more detail a bit later but for now let's just say that by disabling compression we trade temporary disk space usage for speed:
Something is wrong here. Which compression algorithm is being used here and how much has it been tuned? A core hypothesis of the likes of zram is that disk access is so slow (even NVME), that you can often beat it with the bit-rate of decompression.
1. Is something slow like gzip being used?
2. Is the compression effort over-tuned for size? Do some space benchmarks and make sure that you aren't saving a few dozen MB on GBs of data.
zstd, with 1-3 effort (you may even find negative is a overall win), and a trained dictionary (your data does all look identical) is probably a good start.
Great that there are performance benefits, but one thing that I think both meson/ninja and cmake (or cmake/ninja) got wrong, and GNU configure got right is "./configure --help". Why do neither of these tools simply add --help here? Yes, their syntax is different, but the issue is not only about --help. Often I could disable documentation or man page via --disable-man or --disable-doc or something like that. I recently had a discussion with a guy who transitioned into ninja, and I reasoned that there should be an option to skip installing man-pages. He thinks everyone needs manpages. I told him I never look at any local man page ever; I only look online for help. And have been doing so for almost 30 years. I understand the 1970s era of man-pages, but I have no use for them (I do gather local documentation, just not via manpages). He did not want to add a way to install his software without having available xmlto, docbook variants (which are a pain to install, just look at the LFS/BLFS instructions and even then they do not work cleanly). Meson and cmake are better than the GNU autotools, but the few things GNU autotools got right, were lost in both meson and cmake, which is interesting. It seems we can only select between different trade-offs here. New is not automatically better. And being 5% faster really is pointless if functionality is removed or not available anymore.
Autoconf self-documentation isn't bad, but CMake used properly runs circles around it. CMake cache variables are individually settable and come with types and docstrings. The intent is that you can use a variety of interfaces, including CLIs, TUIs, and GUIs to edit them.
Sadly, most people use CMake wrong. (They should read https://cliutils.gitlab.io/modern-cmake/) But still, when used as intended, CMake configuration is a joy to use. Try cmake-gui and ccmake.
[Ninja author here] Nice post, cool to see the deep dive! I also appreciate the details on how they produced their numbers.
As they observe, Ninja gets to be fast mostly by cheating: it avoids a lot of work by saying many things are just out of scope for Ninja to do, and that means it is a useful a target to race against. (Funny thing: when I wrote Ninja I was misremembering how fast an earlier build system was so I kept trying to make it faster. So don't treat it as a lower bound, I just made it up!)
I comment here to say I find the explanation for 'why' in this post unsatisfying. They mention three design decisions.
The first one is a criticism of CMake, not Ninja (?), so I don't think it can be why. I might have misunderstood?
The second reason given is doing some work like header dependencies in multiple threads. This is the most plausible reason to me but it still feels unlikely. It's a very small amount of work: the post mentions 300 compiles, so maybe parsing 300 small text files?
The third is that they run the compiler up front an additional time to gather headers, which is strictly more work than Ninja. There is some hand waving about file access patterns but I am skeptical; if the end-to-end build time is 3 seconds then the project is small enough to all fit in kernel caches. They also mention doing other things like invoking the compiler to get version information. This seems like it would dwarf any performance gain from number 2.
Maybe it's just my own curiosity, I think this post would be better if it had a better explanation for the reason. I'm not disputing the result, I just think the result should make you suspicious that something else is going on, and you might learn something from that! You could for example explore whether it's the header dependency thing by profiling the Ninja invocation and seeing if it's waiting for CPU or waiting for tasks to execute.
(If I had to guess without looking at any of the involved code, I would predict it's something about how CMake generates the build, like it introduces serialization in a place where build2 is parallel, or it adds some extra build steps like gathering the current git hash into a header file or something.)
Simple CMake projects using the Ninja generator are very efficient, unless you create generated files.
And if they are in their own targets, they are not really an issue (they would serialize everything that depends on them as you'd expect), but if you have them in a library grouped with other files to compile, then the whole library compilation is serialized.
And obviously worse if you also have to build the generator for the generated files, but that's not a big surprise, you can observe that in full builds of Chromium or its libraries too waiting for protoc if you crank the parallelization a lot.
Yes, I don’t remember the details but vaguely remember that CMake tends to group things together that could in principle be made more parallel, as you mention with generates files in a library. On the other hand if build2 makes it easier for authors to express these kinds of patterns without the serialization then I count that as a win for build2!
Always glad to see authors on here. Thanks for your insight!
Since they had to rewrite the build file for their program I also assume that something is missing. Didn’t see any mention of verifying that.
I also really didn’t like their denigrating tone. It totally turns me off trying build2, because it seems they don’t understand the point of separating build stages like environment setup (getting dependencies), configure, native build, cross build, packaging. I am a very happy ninja user instead of a batteries-included solution because it does its one job well and can be used very flexibly. I personally detest cmake, so I use nix + own configure script + ninja.
The blog is also wrong about ninja being unable to call configure, but I intentionally don’t want that (I want the build stages to communicate in one direction for sanity).
> I want the build stages to communicate in one direction for sanity
That's easy to do if you control the whole pipeline and can integrate all the features together, not so much with the CMake model unfortunately. I think it would be nice if CMake had Ninja integrated as a library, it could lead to some nice optimizations later.
I wish that when people posted performance results, they would include a theory explaining them. Unexpected performance wins are often noise, bugs, or harness issues. No program is faster just because of the superior manly virtue of its programmer.
Every effect has a cause, and when it comes to performance, it's crucial you understand that cause. If (as in this case) something, from first principles, shouldn't be faster, yet it is, you dig and understand, not just say your shit rules and the other thing sux0rz.
> Let's see if we can go even faster. Next, we disable compression in the file cache. We will discuss the file cache in more detail a bit later but for now let's just say that by disabling compression we trade temporary disk space usage for speed:
Something is wrong here. Which compression algorithm is being used here and how much has it been tuned? A core hypothesis of the likes of zram is that disk access is so slow (even NVME), that you can often beat it with the bit-rate of decompression.
1. Is something slow like gzip being used?
2. Is the compression effort over-tuned for size? Do some space benchmarks and make sure that you aren't saving a few dozen MB on GBs of data.
zstd, with 1-3 effort (you may even find negative is a overall win), and a trained dictionary (your data does all look identical) is probably a good start.
Great that there are performance benefits, but one thing that I think both meson/ninja and cmake (or cmake/ninja) got wrong, and GNU configure got right is "./configure --help". Why do neither of these tools simply add --help here? Yes, their syntax is different, but the issue is not only about --help. Often I could disable documentation or man page via --disable-man or --disable-doc or something like that. I recently had a discussion with a guy who transitioned into ninja, and I reasoned that there should be an option to skip installing man-pages. He thinks everyone needs manpages. I told him I never look at any local man page ever; I only look online for help. And have been doing so for almost 30 years. I understand the 1970s era of man-pages, but I have no use for them (I do gather local documentation, just not via manpages). He did not want to add a way to install his software without having available xmlto, docbook variants (which are a pain to install, just look at the LFS/BLFS instructions and even then they do not work cleanly). Meson and cmake are better than the GNU autotools, but the few things GNU autotools got right, were lost in both meson and cmake, which is interesting. It seems we can only select between different trade-offs here. New is not automatically better. And being 5% faster really is pointless if functionality is removed or not available anymore.
Autoconf self-documentation isn't bad, but CMake used properly runs circles around it. CMake cache variables are individually settable and come with types and docstrings. The intent is that you can use a variety of interfaces, including CLIs, TUIs, and GUIs to edit them.
Sadly, most people use CMake wrong. (They should read https://cliutils.gitlab.io/modern-cmake/) But still, when used as intended, CMake configuration is a joy to use. Try cmake-gui and ccmake.
> And now we are 2.2% faster than Ninja!
This claim isn't supported by the author's measurements?
1 - 3.355/3.429 = 0.021581 = 2,2%
no?
At least you know it was written by a fallible human. But if you point out my mistake, I will be happy to fix it.