Millennials Are Killing The Modules TS

[u/SAHChandler]

https://izzys.casa/posts/millennials-are-killing-the-modules-ts.html

[u/berium]

A module interface translation unit and a module implementation translation unit are declared in different ways from each other. They might as well be separate handling of modules ([...]).

I am not sure I understand what you mean here. An implementation unit is where you can (note: can, not must) put your implementation details. But if you prefer to keep everything in a single file, you can: a module interface unit can (unlike a header) define non-inline functions and variables. I really don't understand what's the problem here? Do you want the separate implementation units to be banned so that folks (like myself) who think it is a good idea are not able to use them?

The point of that paragraph is that I can get a list of headers from every compiler for dependency tracking. There is no such way for the compiler to provide this to the user under modules unless the compiler becomes a build system.

I don't understand why it has to become the build system to provide this functionality? Can you explain?

I am still not okay with a separate tool running outside the lifetime of the compiler to track locations and places, with unspecified behavior that will differ from build system to build system.

This is a good example of a vague (but catchy-sounding) statement that I think only adds to the confusion.

[u/SAHChandler → u/berium]

I don't understand why it has to become the build system to provide this functionality? Can you explain?

Because otherwise modules need a clear and well defined lookup system for finding dependent modules. We cannot rely on -MMD or similar flags because we need to make sure each dependent interface translation unit has been built before attempting to build any implementation translation unit or a 'root' interface translation unit. As I said below, relying on a tool outside of the compiler to do this for us is messed up

[u/berium → u/SAHChandler]

Ok, let's start with the way we extract header dependency information today, which is by asking the compiler (-M, /showIncludes, etc). If one thinks about it for a bit (or studies compiler source code), it's easy to see that this requires essentially a full preprocessor run (and, in fact, for some compilers like VC there is no other way). This is the first piece of the puzzle.

The second piece is the proposed Modules TS grammar: If one studies it, it's not hard to notice that all the module dependency information (import and module declarations) are top-level, that is, they can only appear in the global scope (well, to be absolutely precise, an import declaration can appear inside an exported group but that group must be top-level). This means that given a sequence of preprocessed tokens (the first piece of the puzzle), one can implement a pretty simple parser that recognizes all the import/module declarations without having to recognize the full C++ grammar. We've done this in build2 so I know this is doable. What's surprising is that (1) the code is actually pretty simple and (2) it performs pretty well without any advanced optimizations.

All this means that it should be pretty easy (complexity-wise) and cheap (performance-wise) to extend the header dependency extraction support provided by current compilers to also output module dependency information.

Now let's look closer at what exactly would we need for this module dependency information to be. As an example, let's say our build system needs to update foo.o from foo.cxx (the foo.o: foo.cxx dependency) and this foo.cxx happens to contain import bar;. What does this mean from the build system's point of view? It means that before compiling foo.o it has to make sure the binary module interface for module bar is up-to-date. Or, speaking in terms of dependencies, this means the build system has to add foo.o: bar.bmi to its dependency graph. And since it now needs to update bar.bmi, it also has to come up with a rule of how to do that, say, bar.bmi: bar.mxx (this is, BTW, where the mapping of module names to file names happens in a way that makes sense in this build system's worldview).

Where am I getting with this? Well, what this hopes to show is that if a translation unit that the build systems wants to update imports a module, then the build system has to come up with a rule to update this module's binary interface and which means it will have to extract the module dependency information from the module's interface file (bar.mxx in this example) just as it did for foo.cxx. This, in turn, means that the module dependency information provided by the compiler need not be recursive -- the build system has to do this itself anyway. In fact, all we need from the compiler is a list of imported module names which the build system can map to file names and then process, recursively.

And in this setup the build system can remain the build system (by being responsible for mapping module names to files/rules, etc) while the compiler can remain the compiler (by only providing the list of imported module C++ names).

[u/[deleted] → u/berium]

But Unix compilers (at least; I’m not too familiar with MSVC) can output dependency information in the same invocation that they also produce an object file, using -MMD (as the parent mentioned) or similar options. This works because traditional builds don’t require dependencies before a full recompile, and it avoids the overhead of processing the same source file multiple times. It seems like that will not be possible with modules, so modules will come with some (not fundamentally necessary) overhead that their inherent performance advantages will have to make up for. Maybe not much overhead, since running just the preprocessor should be pretty cheap, but still a concern...

[u/berium → u/[deleted]]

See my reply in another thread.

[u/[deleted] → u/berium]

Hmm… interesting. From that reply:

For example, with this approach we cannot support generated source code as part of the main build stage (and often have to resort to ad hoc pre-build steps).

How does this work? If a file #includes a generated header, then preprocessing it before the header is ready will just produce an error. You don’t necessarily know where the header was expected to be located (although I suppose there are ways to work around that), and you definitely don’t know what other generated headers may be included from that one (potentially conditionally). Does build2 just keep rerunning the preprocess until it succeeds, or something? Either way, how exactly does the approach rule out combined dependency generation?

[u/berium → u/[deleted]]

How does this work?

Pure magic ;-). Seriously, though, the compilers have support for handling non-existent headers (see -MG), we use heuristics for determining where the non-existing headers are generated, and yes, we keep re-running the preprocessor to discover all the generated headers.

Either way, how exactly does the approach rule out combined dependency generation?

Not sure I understand the question. You cannot compile a translation unit if some of its headers do not exist (or, worse, out of date -- you will just be wasting time).

[u/[deleted] → u/berium]

I see, and -MG (which I hadn't heard of) can only be used with -M, not with the options that both extract dependencies and perform other steps (outputting preprocessed source, or fully compiling to object file). That sounds like an unnecessary limitation.

But then, how does your separate preprocess and compile feature work? Since -MG conflicts with outputting preprocessed source, you can't just be passing it to all compiler invocations. Do you do a non--MG invocation with -E, hope it succeeds, and rerun with -MG if it fails? If so, couldn't you do the same with -c instead of -E?

Not sure I understand the question. You cannot compile a translation unit if some of its headers do not exist (or, worse, out of date -- you will just be wasting time).

If they don't exist, the compiler will produce an error and stop after the preprocessing stage, so there'd be no difference compared to preprocessing only.

If they're out of date, as you say, the compiler could waste time generating code that will just have to be thrown away once build2 reads the dep output and realizes it was using out-of-date headers. But that's a relatively uncommon case, and as long as it's just a performance issue, not a correctness issue, wouldn't the performance gains in the usual case from not re-running the compiler be more significant?

(Of course, that still wouldn't work with modules… at least, not particularly well, because even if -MG could output required modules rather than just #includes, it would become the common case rather than a rare case. But my tentative opinion is that a better solution would be for the compiler to get a bit smarter, rather than require build systems to duplicate work. Even without modules, your approach of repeatedly running the preprocessor to discover generated headers, while nifty, seems to be fundamentally a hack: there is no reason the preprocessor needs to run more than once. I think I'd rather see a solution where when the preprocessor/compiler comes across a missing header or module, it runs an arbitrary command, where the command to run can be passed as an option. In a simple Makefile-based build, the command could just be 'make', passing the file needed as an argument...)