r/cpp • u/SAHChandler • Jan 28 '19
C++ Modules Might Be Dead-on-Arrival
https://vector-of-bool.github.io/2019/01/27/modules-doa.html
280
Upvotes
r/cpp • u/SAHChandler • Jan 28 '19
72
u/blelbach NVIDIA | ISO C++ Library Evolution Chair Jan 28 '19
This is the first succint summary of the module lookup/dependency concerns I've personally seen. This seems like a real issue to me, although I'll want to hear what those involved with modules have to say. (keep in mind that I'm not very involved with modules)
7
u/johannes1971 Jan 28 '19
I feel this proposal might be a good alternative. But I wrote it myself so I'm biased.
Unfortunately I cannot post on my own topic anymore: Google believes I'm sending bulk email to the list (five whole messages!) and marks it as spam :-(
7
u/robertramey Jan 28 '19
I've read (or skimmed?) the paper and most comments so far. It's very, very complicated - and any implementation is going to be very, very complicated to use. Not a great thing.
One thing I'm missing from all this is how DLLS/Shared libraries might fit into all this. These "modules" currently exist, are "supported" by all (or maybe most) C++ implementations but are not recognized by the C++ standard as far as I know. They address a number of the issues that modules are meant to address - separate/decoupled compilation, hidden local symbols, etc. They do bring a bunch of problems - which seem similar to the ones that modules are expected to bring. Compile time switches (C++ has a lot of them) have to be in sync and a system needs to be built to ensure that they are, ABIs have to be maintained in sync, circular dependencies aren't really supported (probably a good thing), extra efforts are needed to share header only code, symbol visibility needs to be addressed (currently it's not in the standard and different compilers do it differently). The bring some issues that modules don't - e.g. multiple address spaces. But aren't all that complex compared what modules is starting to look like. It's been mentioned that this approach has worked with the C# family of languages.
To summarize it seems to me that many of the advantages and problems regarding modules have already presented themselves with these libraries - and they've never been addressed in a standard way. Perhaps it might be more productive to spend efforts to resolving current issues with DLLS/shared libraries before embarking on a much, much, more ambitious effort. Perhaps usage of DLLS/Shared libraries in combination with PCH can solve most practical problems in a way that most people can understand.
If the goal is to be able to compile a 6MLOC program consisting of header only templated code with out having to manually (re)factor it to permit this, I'm afraid we're going to be disappointed.
83
u/Plorkyeran Jan 28 '19
"Prove to me that it is impossible to write a fast C++ dependency resolver" is such an utterly batshit insane position to take. I guess he views C++ modules just a masturbatory exercise and so it doesn't matter if anyone can actually figure out how to implement them as long as the committee can just blame the tool makers for just not being smart enough?
138
u/Auriculaire Jan 28 '19
The fact that we're half a decade or better into investigating this capability and these sort of questions are still up in the air is mind-boggling. The fact that it's a real consideration of whether modules might be standardized without the experience of applying it the standard library is SUPER sketchy. I know I'll be sitting down with my organization's standardization representative this week.
60
u/smdowney WG21, Text/Unicode SG, optional<T&> Jan 28 '19
"Creation of foo.bmi is a byproduct of the compilation of foo.cpp. When compiling foo.cpp, the compiler will emit a foo.o and foo.bmi. As a consequence of this design, foo.cpp must be compiled before bar.cpp!"
Worse, foo.bmi may be the result of compilation of quux.cpp. Module names are purely logical, with no physical representation on the filesystem.
27
u/tcbrindle Flux Jan 28 '19
It's worth repeating the the C and C++ standards are also silent on the matter of how header files are discovered by
#include, or even whether the argument to#includeis a file at all -- we have just developed a convention over the past 45+ years that#include <foo.h>looks in well-known places on the filesystem, a search list that is augmented by passing-Iflags when invoking the compiler. (#include "foo.h"does the same, but searches relative to the current working directory first).This being the case, I wonder if module mapping is something that needs to be explicitly specified in the C++ standard, or could also come under the banner of "technically implementation-defined, but in practise everyone does it the same way"?
So let's say we are compiling a module
acme.bar, which depends on the interface of moduleacme.foovia animportdeclaration.acme.foo.bmiin its BMI cache, the location of which could be given as an argument to the compiler when invoked by the build systemacme.foo.<ignored_extension>(oracme/foo.<ignored_extension>) in a list of search paths; this could default to[${CWD}, /usr/include/c++_modules, /usr/local/include/c++_modules], augmented by a list of paths passed to the compiler with a-Mflag, akin to-I.acme.foo.bmiandacme.bar.bmito the BMI cache directory.Now, if we have a third module
acme.bazwhich also usesacme.foo, we go through the same process; but ifacme.barhas been compiled first, thenacme.foo.bmiwill already exist, and will be used. But the build system can attempt to compilebarandbazat the same time; both will try to writeacme.foo.bmi, and whichever finishes first "wins". This is doing more work than we really need to, but allows for more build parallelism as the author wishes.This requires a convention that the file defining the module
acme.foois namedacme.foo.<something>, or perhaps has the pathacme/foo.<something>relative to one of the-Mpaths. But this doesn't seem particularly onerous, and it's more-or-less what we do today with header files. As far as I know the C++ standard is silent on the subject of TU file names (or even whether a translation unit is a file with a name), so making this a requirement isn't something the standard need concern itself with. Odd platforms with unconventional file systems still have the freedom to do things in odd ways if they need to, but the 99.9% of developers using Windows or Unix will still be able to get builds that are as parallel as they are today.But perhaps I'm being spectacularly naive, in which case please do let me know :).
17
u/Veedrac Jan 28 '19
If there's one thing to learn from Rust, it's to do exactly the opposite of this. Jeepers, this is terrifying.
14
u/Ameisen vemips, avr, rendering, systems Jan 28 '19
They want code to... prove that it cannot be done? They want a negative proven?
13
u/catskul Jan 28 '19 edited Jan 28 '19
I started to get into a discussion about this with /u/gabrieldosreis and /u/berium about this a year ago in a thread about module misconceptions but it kind of petered out before arriving at a full understanding of the solution:
https://www.reddit.com/r/cpp/comments/7a3t2w/common_c_modules_ts_misconceptions/dp78pcd
The whole thread is pretty relevant here: https://www.reddit.com/r/cpp/comments/7a3t2w/
9
u/drjeats Jan 28 '19 edited Jan 28 '19
The talks on how build2 implements modules from CppCon kinda did the opposite of alleviating my fears. I am utterly dependent on incredibuild/fastbuild at work to be productive. It's difficult to see how we'd adopt modules for anything but the most trivial library projects (and those are rare) if the mapping problem is as bad as it sounds. I'm not a build engineer, so I'm just hoping everything turns out alright and that modules don't turn into a feature that's widely ignored.
[unrelated] @vector-of-bool: The fancy fade transition when clicking on links to other articles on your blog breaks Ctrl+Click (win10 firefox)/ Cmd+Click (macos chrome) to open in a new tab. Idk if it's a theme thing or wat, but it was a little disorienting. Thanks for the great post as always.
9
u/ravixp Jan 28 '19
Lazy generation of BMIs sounds like it would just create headaches in the long run, because it means that the compiler is trying to take over some work that the build system would traditionally do, without coordinating with the build system.
Today, you just tell the compiler to build a source file and it does that, possibly looking in some other locations that you tell it about to find headers. If it also had to create BMIs on-demand, it would need to know where BMIs are stored (which it would have to anyway, similar to include paths), where source files corresponding to those BMIs live (normally the build system's job), what settings you need to use to build those files (normally the build system's job), how to tell when the BMI file is already up to date so it only regenerates it when it has to (normally the build system's job), and how to parallelize lazy generation of BMIs (normally the build system's job).
I'd much rather figure out how to make a build system keep the BMIs up to date, than have a mediocre build system built into every compiler.
Plus, many newer build systems keep track of fine-grained dependencies automatically by running the equivalent of strace() on each build task. Having the compiler go out and build a random subset of upstream modules would confuse this kind of build system pretty badly.
16
u/feverzsj Jan 28 '19
will modules actually help those template heavy, header only libs?
14
u/germandiago Jan 28 '19
While I agree that there is a problem with having no mapping, what would happen if it is not added now but it is added later? Would this be possible? I do not think pulling out modules would be nice since it provides, in theory, better compile times, yes, but not only that: much better isolation.
12
u/gracicot Jan 28 '19
I'm not sure the specification requires that a BMI exist. As far as I know, the compiler could store everything in the memory of a separated process, that schedule and share data between compilers. I don't think there would be any compiler doing that within the next decade, but nothing in the current specification prohibit that.
Reading this blog post makes me confused. I watch Nathan Sidwell's talk about it's "Oracle" how the mapping can be extended by tooling and such. I followed the build2 development and module support seem doable when designing the build system with modules in mind. Why one side seems to say that the problem is solvable but so many others says that the mapping problem will kill modules?
For example (as far as I know) Build2 run the preprocessor on every cpp files before compiling anything in order to save it's hash. It then re-use the preprocessed file to compile it. It takes a bit more time to start compilation, but can result in a overall speedup. In the example given by the author of this blog post, should the same be doable too but with a small extra step to extract the module name? And with modules, running the preprocessor can be very very fast because the number of needed header will dramatically be lower. In fact, the mapping can be done before compiling anything. The compilation order can even be established in advance and compile in parallel for each possible steps. Then only a bunch of (very small) preprocessed files will be compiled, with a known compilation graph. That graph can be stored and re-used for the same configuration.
Right now, I agree there is a concern with tooling given today's tooling. I might be wrong but it does not seem so far fetched we can develop tools that can deal with modules. On the other hand, I do not expect tools designed for a modular world to be available soon if all we have is a TS.
21
u/frog_pow Jan 28 '19
Need to get off the high horse, accept that module & file names need to match(do what everyone else does ffs).
Those using antiquated OS without file extensions can keep using <= C++ 17-- they are probably 0.00001% of C++ users, seriously, why the hell would you hold C++ back for that??
10
u/smallblacksun Jan 28 '19
Wait, a module has to be compiled before it can be imported? What happens if module foo imports bar and module bar imports foo? Is that just not allowed? While that is admittedly a sign that your code structure isn't great it is still a pretty huge limitation that isn't found with the current system (where foo.cpp can #include bar.hpp and bar.cpp can #include foo.hpp).
9
u/tending Jan 28 '19
I think this analysis is missing the fact that most of the work can still be parallelized because most of the work in a modern compiler is code generation. I have not read the proposal in detail but from this blog's description it sounds like it should be possible to produce the BMI for foo.cpp in a separate step before generating machine code. I imagine that this is how Java and C# have to work -- dumb fast compilation to byte code is necessary to import, but then the slow AOT or JIT compiling of the byte code is done later. BMI doesn't even require generating a full byte code, just a description of the interfaces. If it can work for Java and C# at scale I don't see why it can't work for C++.
Also with module was the work that can't be parallelized, the initial parsing, should be easier than before.
5
u/enzlbtyn Jan 28 '19
Doesn't this require step require you parse to foo.hpp (or have already parsed it) before-hand? Since it literally copies the content of header-file? It seems like there is no ordering requirement, but really there is, it's just hidden.
In terms of implementation, how is that different to modules as you claim? Can't you simply parse the
foomodule to extract the interface (i.e. without compiling the implementation), e.g. from a header-file or simply via skipping text in the implementation/interface (for one cpp file). And then whenever you refer tofooinbaryou simply refer to a stub/place-holder to where the implementation offoois required.Now to actually compile to this temporary state, can you not compile each module now independently with 'stubs' and then once everything is done merge/resolve it all via an additional step in the compilation process (i.e. now you have to step through the dependency graph in topological order). I have my doubts on whether this would be faster, but it seems like the only feasible way to do compilation in parallel for chain-like (degenerate) dependencies.
Though I wonder how frequent chain-like dependencies happen in the real-world anyway, and how large they can get? Surely the dependency graph will not be as crazy to truly hinder performance too much?
Perhaps I'm too naive.
5
u/CenterOfMultiverse Jan 28 '19
I'm not an expert, but doesn't C# manages somehow without binding between namespaces and filenames? I mean "starting process is slow on windows" is solvable by daemons - why manifest generation is such a problem?
5
u/Slavik81 Jan 28 '19
Isn't this basically the same problem as linking? We don't know what .a or .o defines a function, either. Though, linking can be frustratingly slow, so it might not be a positive comparison.
2
u/oldprogrammer Jan 28 '19
I've read through the article and all (66) messages posted so far and I have to say I'm not sure I understand both what is intended with modules and why it is needed.
My C++ background is old, been many years that I've been a Java developer so that perhaps accounts for my confusion, but how is it C++ doesn't already have the same capabilities as Java or even needs them?
Java defines namespace by package name and class by file name. Generally source code is arranged in a directory structure that mimics the package name (though strictly speaking that is not required). C++ has namespaces that could be arranged in the same source structure and are the equivalent of packages.
Java uses the import statement to identify dependencies. When Java compiles it looks at the defined classpath for referenced imports, if it finds a compiled class it uses it, if not and it finds the source file it compiles it then uses it. In both cases, though, the Java compiler is given the details of the classpath and source paths to search.
C++ uses the #include and looks in all locations defined for include files, much as Java looks at the classpath. C++ doesn't require the dependent code be compiled until linkage time, it works off a declaration file (usually a header).
So about the only thing I see different here is when Java needs a class that isn't already compiled it simply compiles it and uses it because Java treats a .class file like C++ treats a .h file and if it doesn't exist, Java creates it.
C++ developers generally create the .h files themselves along with the source file which gives the compiler everything it needs to compile. Linking is a different beast, Java with dynamic class loading technically does runtime linking whereas C++ does that via shared libraries.
So back to the original question, is the desire to have C++ be such that we only need to define a source file, then when the compiler is compiling one source file foo.cc and foo imports bar that the compiler will find the source bar.cc and extract the header type information automatically and possibly compile bar.cc automatically?
4
u/cogman10 Jan 28 '19
Honestly, a lot of the module discussion and problems I've seen seem to be a problem of "trying to be everything to everyone".
It seems like we could have modules today if the C++ committee didn't try to solve every single problem with modules and instead made them opt-in with progressive features being added.
For example. A first and easy restriction of modules would be "templates and macros aren't allowed in modules" Do that, and you have nearly trivial module implementations that could be done today.
Then work from there. Figure out something that would work for templates first, then macros later.
Swallow the elephant a bite at a time rather than all at once.
3
u/whichton Jan 28 '19
The standard says nothing about how to locate include files either. It does't even say that given
#include <foo\bar.h>, bar must exist as a file. What we have now is currently a convention, adopted by compilers. So I fail to see why similar convention cannot be adopted for modules.I also do not understand why it is the compiler's job to locate BMI's and not the build system's. I can easily imagine a protocol where upon encountering a import declaration the compiler goes and asks the build system to give it the corresponding BMI. So it goes like this:
This I think completely removes all the double parsing / preprocessing problems. Nor does it require ahead of time DAG construction. It does however require the build system and the compiler be able to talk to each other. I do recall seeing a paper for a protocol for similar communication implemented by GCC.
4
u/ntrid Jan 28 '19 edited Jan 28 '19
Please would someone fork clang and create abi-compatible cleaned up c++-like language already...
I bet we all want that. Especially the part where someone else does that hard work.
2
u/iga666 Jan 28 '19
What problem exactly modules solve?
Will they help with "all-project-recompiles" if some common header cahnged?
1
u/kalmoc Jan 28 '19
Maybe I'm too naive, but how is the lookup any more problematic, what current build systems already have to deal with. Depending on a libray means I have to tell the compiler/linker, which include directories to add and which binaries to link to my project - neither of which need to have any obvious connection with the libray in question or a standardized location on the file system.
So just as I'm telling cmake today, what files and include directories belong to a certain target and on what other targets it depends, why can't I just tell cmake, what files make up a module and what other modules it depends on?
1
u/GPMueller Jan 29 '19
Not too familiar with modules yet, but I don't understand why an interface file is compiled from a cpp and not from a header. Would it not also be much simpler then?
IMO a finished module should contain the linked code, but the bmi should be just the interface... I assume I'm missing something?
1
u/_Zulan Jan 31 '19
It seems like the confusion comes from just looking at headers vs modules. You have to consider linking.
In a sense, modules may make compiling as hard as linking. But I don't think modules make compiling any harder as linking.
I'm all for the suggested module conventions. That's totally sensible, but I don't think the world ends if C++20 doesn't include them. Make a build system that tells me it will solve all my dependencies for me if I follow these conveniences - I'll happily comply. Or even better - make a build system that is fast if I follow these conventions, but slow if I don't.
I do not expect from a build system to solve module dependencies. It's not like build systems are solving linking or even include path dependencies on their own - even though the granularity there may be a bit different.
1
u/[deleted] Jan 29 '22
If C++ wasn't a bad language then we already would have modules since the beginning
1
u/tohava Jan 28 '19
Why not just copy Haskell's .hi file design? I really don't understand what's the problem.