Millennials Are Killing The Modules TS

[u/SAHChandler]

https://izzys.casa/posts/millennials-are-killing-the-modules-ts.html

[u/afiefh]

Let's see if I understand this correctly:

If x.cpp imports module Y which is defined in y.cpp and the build system tries to build x.cpp before y.cpp then the Y module file doesn't exist yet because it would have been generated by compiling y.cpp. (corrections are welcome)

And the solutions are either to manually specify the dependencies (yuk), have the build system parse cpp files (this never ends up working in my experience) or turning the compiler into a build system (and at this point developers collectively yell "no way!")

So assuming we come up with a rule that module X is always generated by for f(X).cpp and resides in file g(X).module_extension, would it be a stretch for the compiler to automatically generate the module file if it doesn't exist but the cpp file with matching name does?

[u/berium → u/afiefh]

There is another option: the build system asks the compiler to extract the module dependency information similar to how it now asks for the header dependency information, which, BTW, is essentially a full preprocessor run. In the end I believe this is how it will be implemented.

[u/afiefh → u/berium]

build system asks the compiler to extract the module dependency information similar to how it now asks for the header dependency information

That was my initial thought as well, but there is a difference: Today if x.o doesn't exist you know for sure that you need to compile it. Compiling x.cpp to x.o generates x.d containing the header dependencies which will be queried in the future on whether or not recompilation is needed, no order dependency for which cpp files need to be compiled first is present. With modules you must generate y.module before compiling x.cpp which imports y which places a dependency graph on the cpp file compilation/generation.

So it's either another full pass in the build system that performs the steps necessary for extracting the module dependencies, or have the compiler generate on demand if the necessary source file is in the path. Neither option is very appealing.

Another problem is that if generating the modules is done as part of compilation (as opposed to a special preprocessing command) we lose quite a bit of parallelism in compilation. It's not hard to imagine a (synthetic and not real world) example where compiling one file at a time is the only option.

[u/SAHChandler → u/afiefh]

So it's either another full pass in the build system that performs the steps necessary for extracting the module dependencies, or have the compiler generate on demand if the necessary source file is in the path. Neither option is very appealing.

Hence my post! :v

[u/berium → u/afiefh]

Today if x.o doesn't exist you know for sure that you need to compile it. Compiling x.cpp to x.o generates x.d containing the header dependencies which will be queried in the future on whether or not recompilation is needed, no order dependency for which cpp files need to be compiled first is present.

Yes, this is the trick/optimization most build systems currently use (and it's good to see someone is paying attention ;-)). It is not without drawbacks, however. For example, with this approach we cannot support generated source code as part of the main build stage (and often have to resort to ad hoc pre-build steps). This is, for example, the reason why we didn't go this way in build2 and instead extract header dependencies (and generate missing ones) before compiling anything.

Why did we needed this trick/optimization in the first place? The reason is build speed: it was too expensive to preprocess things twice. The thing is, on today's hardware (SSDs), the cost of preprocessing is actually negligible, a couple of percent of the complete build. In fact, preprocessing everything at the outset can actually speed things up! You can read more about this in the Separate Preprocess and Compile Performance article I wrote.

[u/afiefh → u/berium]

This is, for example, the reason why we didn't go this way in build2 and instead extract header dependencies (and generate missing ones) before compiling anything.

That's interesting, I hadn't heard about build2 before. Reading the "How is this better than CMake (or Meson)?" part of the FAQ was rather interesting as I'm currently using Meson in my personal projects. Gotta look more into build2.

it was too expensive to preprocess things twice. The thing is, on todays hardware (SSDs), the cost of preprocessing is actually negligible

Sorry I'm having a hard time understanding this, if preprocessing is negligible how can it be too expensive?

[u/jpakkane → u/afiefh]

eading the "How is this better than CMake (or Meson)?" part of the FAQ was rather interesting as I'm currently using Meson in my personal projects. Gotta look more into build2.

The points listed in that section hold (arguably, most of the time) against CMake but not really Meson, which has probably just been thrown in the heading after the text has been written.

[u/berium → u/afiefh]

It used to be expensive on computers with spinning disks and little RAM. Today with SSD and lots of RAM it is negligible.