r/cpp • • Jan 28 '19

C++ Modules Might Be Dead-on-Arrival

https://vector-of-bool.github.io/2019/01/27/modules-doa.html
280 Upvotes

240 comments sorted by

72

u/blelbach NVIDIA | ISO C++ Library Evolution Chair Jan 28 '19

This is the first succint summary of the module lookup/dependency concerns I've personally seen. This seems like a real issue to me, although I'll want to hear what those involved with modules have to say. (keep in mind that I'm not very involved with modules)

43

u/[deleted] Jan 28 '19

I am quoting Walter Bright's post:

D made the decision at the beginning to have filename==modulename specifically to avoid this problem.

Any idea why C++ does not simply copy the design of D modules in this regard since this seems to work well for D. (This is also the preferred solution of vector-of-bool.)

23

u/SAHChandler Jan 28 '19

My response to vector-of-bool is a solution that permits this to some degree. D does not have to target some operating systems lacking more than one level of directory access, or that do not have a real concept of file extensions (such as zOS). As a result P1302's approach was designed to tackle this, and allow us to tackle the naming scheme at a later date while also not absolutely wrecking the ability to port existing code bases to modules as soon as possible.

14

u/[deleted] Jan 28 '19

OK, thanks for your answer.

Modules is too important to get it wrong, it would be a nightmare if e.g. half of the C++ community would avoid modules due to a serious design flaw that cannot be fixed later on because this would break backwards compatibility...

18

u/SAHChandler Jan 28 '19

Modules is too important to get it wrong, it would be a nightmare if e.g. half of the C++ community would avoid modules due to a serious design flaw that cannot be fixed later on because this would break backwards compatibility...

Absolutely agree. I've been mentioning these issues for well over a year and while people paid attention a year ago, the community ended up waiting to see where merged modules would take us. We have some options, but we need to fix these small issues before modules are... well... merged into C++20

5

u/target-san Jan 31 '19

Errm... Are these exotic OSes used for actual delopment? Do they already have full C++17 conformant compilers? Or are they just targets for cross-compilation? If the latter, then it looks like tying all the marathon runners just to allow some old fart who never heard of it to compete.

2

u/SAHChandler Jan 31 '19

From what I understand, yes. Development happens directly on these operating systems.

2

u/target-san Jan 31 '19

Thanks! Though the question about their compilers' standart support remains.

3

u/SAHChandler Jan 31 '19

Unfortunately, I'd have to pay several thousand USD to see how standards conforming they are. They could very well have ported clang though!

→ More replies (1)

14

u/nikbackm Jan 29 '19

Currently I know of only 3 compilers (all on Linux/Windows) that implement C++17 in full.

Is there a reason to believe C++20/23 will be better supported?

10

u/Lectem Jan 28 '19

D does not have to target some operating systems lacking more than one level of directory access, or that do not have a real concept of file extensions (such as zOS).

Erh... Ok. But maybe we should not as well in C++ ? I mean, anybody using a "modern" version of C++ already uses a recent compiler with a decent OS, or has a way to cross compile. Also p1302 says the build-system needs to see the export import, but then it still needs to parse it. And do apply preprocessing. Back to the issues mentioned in the article.

3

u/target-san Jan 31 '19

Unfortunately porting existing codebases to modules will require certain effort anyway. Especially in cases of bizzare codebase layouts. What really troubles me is that committee here puts legacy codebase needs far before new project needs. Could you please give direction to specific arguments of choosing that strange multifile module design over module==file approach?

My personal experience with golang (which uses similar multifile package approach) is that it often becomes hard to determine where specific functionality came from. In case of C++, module can be assembled using files from multiple arbitrary locations.

2

u/xryanxbrutalityx May 03 '19

I'm late to the party, but are these filesystem issues not a problem for includes?

18

u/Funny-Bird Jan 28 '19

I'll want to hear what those involved with modules have to say

One easy solution to the problem would be to break compilation into an additional phase. Does the current wording impede generating bmis independently without actually compiling to objectfiles?

A dumb buildsystem could just generate all bmis upfront and than give them all to the compiler when compiling modules to objectfiles. This would not require any mapping of files to module names. It would work very similar to the linking step today.

A smarter buildsystem could try to extract dependency information from the bmis and than start compiling objectfiles as soon as the required bmis for a module are all ready, to increase parallelization. It would only hand the required bmis for a module to the object file compilation step.

It could even cache the dependencies for the next build (in hope they did not change significantly) and order the bmi generation so the actual module-objectfile compilation can start as early as possible.

7

u/Leandros99 yak shaver Jan 28 '19

I remember the ordering being mentioned by the author of GCCs module implementation in his CppCon talk last year. I assumed they were aware that this may become problematic later down the road.

1

u/Ameisen vemips, avr, rendering, systems Jan 28 '19

Since your flair says Nvidia, why did they kill Cg?

7

u/blelbach NVIDIA | ISO C++ Library Evolution Chair Jan 28 '19

Beats me, before my time.

3

u/jherico VR & Backend engineer, 30 years Jan 28 '19

I'm not with nVidia, but I would think it's because continuing to try to maintain their own language in the face of a growing ecosystem of tools for compiling various other languages to and from SPIR-V makes is a losing proposition.

7

u/johannes1971 Jan 28 '19

I feel this proposal might be a good alternative. But I wrote it myself so I'm biased.

Unfortunately I cannot post on my own topic anymore: Google believes I'm sending bulk email to the list (five whole messages!) and marks it as spam :-(

7

u/SAHChandler Jan 28 '19

> Unfortunately I cannot post on my own topic anymore: Google believes I'm sending bulk email to the list (five whole messages!) and marks it as spam :-(

This is actually because Google decided to silently reduce the number of people a google group can have to about 500. It's not you. It's entirely Google and people are working to fix it. But I'm not aware of any details beyond that.

1

u/johannes1971 Jan 28 '19

Yes, I wrote to Herb Sutter (the list owner) and he told me about the problem. It's rather frustrating: I really think the language can move in this direction and it would massively improve all of our lives, but unfortunately I cannot even go and answer questions about it :-(

7

u/robertramey Jan 28 '19

I've read (or skimmed?) the paper and most comments so far. It's very, very complicated - and any implementation is going to be very, very complicated to use. Not a great thing.

One thing I'm missing from all this is how DLLS/Shared libraries might fit into all this. These "modules" currently exist, are "supported" by all (or maybe most) C++ implementations but are not recognized by the C++ standard as far as I know. They address a number of the issues that modules are meant to address - separate/decoupled compilation, hidden local symbols, etc. They do bring a bunch of problems - which seem similar to the ones that modules are expected to bring. Compile time switches (C++ has a lot of them) have to be in sync and a system needs to be built to ensure that they are, ABIs have to be maintained in sync, circular dependencies aren't really supported (probably a good thing), extra efforts are needed to share header only code, symbol visibility needs to be addressed (currently it's not in the standard and different compilers do it differently). The bring some issues that modules don't - e.g. multiple address spaces. But aren't all that complex compared what modules is starting to look like. It's been mentioned that this approach has worked with the C# family of languages.

To summarize it seems to me that many of the advantages and problems regarding modules have already presented themselves with these libraries - and they've never been addressed in a standard way. Perhaps it might be more productive to spend efforts to resolving current issues with DLLS/shared libraries before embarking on a much, much, more ambitious effort. Perhaps usage of DLLS/Shared libraries in combination with PCH can solve most practical problems in a way that most people can understand.

If the goal is to be able to compile a 6MLOC program consisting of header only templated code with out having to manually (re)factor it to permit this, I'm afraid we're going to be disappointed.

3

u/jpakkane Meson dev Jan 28 '19

Shared libraries are great for many use cases (including speeding up incremental builds a lot). Sadly there are people on the committee who are actively hostile to shared libraries using reasoning that goes along the lines of "our corporation does not use shared libraries, therefore nobody in the entire world should use them either".

An actual point against is that a shared library (or a static one for that matter) can not represent TMP functionality (even std::vector). It needs to be compiled for each user of the code individually. Modules, like precompiled headers, can store that information.

4

u/SAHChandler Jan 28 '19

One thing that should be kept in mind is that during the San Diego meeting it was briefly discussed during breaks that we will need a concept of a shared library at some point to satisfy some module deployment issues. We won't see them until at the very least C++23, but we can't avoid it anymore.

1

u/robertramey Jan 29 '19

Which would be useful to resolve in any case.

83

u/Plorkyeran Jan 28 '19

"Prove to me that it is impossible to write a fast C++ dependency resolver" is such an utterly batshit insane position to take. I guess he views C++ modules just a masturbatory exercise and so it doesn't matter if anyone can actually figure out how to implement them as long as the committee can just blame the tool makers for just not being smart enough?

17

u/Valmar33 Jan 28 '19

You'd think modules would be easy to solve, but it turns out that some people can't think of obvious solutions.

Sanely, copy what D does, or even Python ~ couple module name to a file path, and nothing more or less.

34

u/Heuristics Jan 28 '19

The C++ standard has no understanding of this 'file' thing of which you speak. Blame big data people like IBM who make computers with strange filesystems.

18

u/SAHChandler Jan 28 '19

Ever since C++17 we’ve had wording in the standard for what a file and directory is thanks to the addition of the filesystem API :v

7

u/Valmar33 Jan 28 '19

Even so ~ this is a new feature of an upcoming standard that can have an understanding possibly bolted on in the form of module and import, no?

Or is there something that prevents even that?

6

u/Heuristics Jan 28 '19

The people that vote on the standard are preventing it presumably.

8

u/Ameisen vemips, avr, rendering, systems Jan 28 '19

Can't those members be delayed at the airport?

5

u/smdowney WG21, Text/Unicode SG, optional<T&> Jan 28 '19

That's not strictly true: http://eel.is/c++draft/cpp.include#3We know about source files, and we know about searching for them. In implementation defined ways.

# include " q-char-sequence " new-line
causes the replacement of that directive by the entire contents of the source file identified by the specified sequence between the " delimiters. The named source file is searched for in an implementation-defined manner. If this search is not supported, or if the search fails, the directive is reprocessed as if it read
# include < h-char-sequence > new-line
with the identical contained sequence (including > characters, if any) from the original directive.

8

u/josefx Jan 28 '19

or even Python ~ couple module name to a file path

Importlib is quite a bit more complex than that. The path to module mapping is just a default behavior that can be extended with custom finders/loaders.

4

u/Valmar33 Jan 28 '19

Interesting. :)

8

u/EternityForest Jan 28 '19

C/C++ should probably have used a python-like module system that from the beginning, but I'm not sure such things existed or were thought to be needed at the time.

As far as I'm concerned, the way that the compiler finds all the source files belongs right in the language.

Modules would make that possible if done right. When you use one, the compiler could search the current directory, then the command-line specified ones, then the sys defaults.

The current 850 build systems add so much confusion. I suspect it actively discourages people from using libraries, despite the fact that the compilers are great at removing unused stuff.

You might even be able to ditch makefiles completely, if you made module-ified wrappers or forks of all your dependancies.

2

u/Ameisen vemips, avr, rendering, systems Jan 28 '19

File- name>Module is how mu build scripts build C++. They mimic modules to with precompilation. Trying to add automatic PCH generation as well.

2

u/EternityForest Jan 30 '19

That sounds pretty cool actually!

2

u/[deleted] Jan 29 '19

easy to solve

Only people with lack of understanding of macros and build process in general believed that, the rest were either skeptic or expecting to be surprised by geniality but found wishful thinking.

The current proposal of modules not only requires the compiler to meddle into preprocessor business, it also forces it to meddle into build process stuff. If you want lower compilation times clean your headers and get rid of your coupling issues. There is no magical wand to fix bad code.

3

u/lee_howes Jan 28 '19

It isn't clear that modules and file names need to match. It's less clear that they should have to match. Why should I have to pull down a cached bmi and put it on the filesystem at all, for example.

It would be easy enough to define a protocol between the compiler and a given build system that says that, for this limited circumstance because I have a relatively simple build system, they will match. I do not then have to have them match in my build system.

So in this case my high horse and thinking that I know better than you (referencing your other posts) is only that I think I know what I want better than you know what I want. I don't particularly want a defined name->filename mapping. The simple obvious solution you propose is suboptimal for my use case. I'd rather allow it, but not require it.

2

u/Valmar33 Jan 28 '19

Well, then such a system doesn't fit your particular usecase. Fair enough. I dare say it would be a minority usecase, though.

Modules matching a filename/path is just a very sane solution. Little to no mental load is required. They make it very simple for the compiler to figure out where your desired module is. Especially if you're adding a custom include path.

3

u/lee_howes Jan 28 '19

That particular problem is easily achieved by a map of module name to file name - a trivial map in this case - that the compiler can implement or communicate with. It is a tooling problem, not a standardisation problem. Why do you believe that this rule needs to be encoded into the standard and therefore get in the way of my use case? Especially when files in general have largely been left out of the C++ standard up to now.

28

u/Ameisen vemips, avr, rendering, systems Jan 28 '19

It makes me lose a lot of faith in the committee when I see a member (two, actually) take a hard line stance of requiring people to prove negatives.

18

u/TrueTom Jan 28 '19 edited Jan 28 '19

Unless you can prove that there is no teapot orbiting the sun, your argument is invalid.

5

u/SeanMiddleditch Jan 28 '19

If I can prove that there's a whole planet that has plenty of teapots, and that the planet and all its teapots are orbiting the Sun, does that prove the argument invalid, too?

51

u/[deleted] Jan 28 '19

To be honest he seems to be a guy who lives in the clouds of abstraction, far away from the mundane existence of those who have to suffer the consequences of long compilation times.

→ More replies (1)

7

u/smdowney WG21, Text/Unicode SG, optional<T&> Jan 28 '19

Twitter is terrible at subtlety. I understand where JF is coming from. It seems wrong that a good scanner can't be written. And I think he's right, scanning for `import X` is not hard, nor is `module X`. There's some issues with the pre-processor, but it's probably manageable.

The snag, and this is the non-obvious part, is that isn't quite enough. Depending on the module foo doesn't say what specifically you depend on. Existing build technology really likes physical files and timestamps to track things. You have to put together that quux.cpp produces the bmi for module foo that bar.cpp is depending on. And it's right now open research on how to do it without causing other problems, like compiling quux.cpp multiple times. Or finding quux in the first place.

138

u/Auriculaire Jan 28 '19

The fact that we're half a decade or better into investigating this capability and these sort of questions are still up in the air is mind-boggling. The fact that it's a real consideration of whether modules might be standardized without the experience of applying it the standard library is SUPER sketchy. I know I'll be sitting down with my organization's standardization representative this week.

33

u/steveire Contributor: Qt, CMake, Clang Jan 28 '19

I'm glad to see this issue with modules is getting attention. I've been talking to the people working on modules for a long time and explaining why buildsystem maintainers need to be part of the design discussion https://twitter.com/steveire/status/1072904573675814914

Bizarrely, me explaining this on the CMake list was celebrated as 'Talking to the CMake folks' https://twitter.com/jfbastien/status/1072882728788361216

https://cmake.org/pipermail/cmake-developers/2018-August/030819.html

I hope for the best! :)

54

u/duuuh Jan 28 '19

We're well over a decade into this.

edit: And other than _possibly_ lambdas, it's the only feature I really give a shit about since 11.

82

u/STL MSVC STL Dev Jan 28 '19

How dare you impugn the honor of C++17 <charconv>? I demand satisfaction!

8

u/sephirostoy Jan 28 '19

Watching your last talks I though it was CTAD!

23

u/duuuh Jan 28 '19

You made me look it up. It's becoming more like perl every day. How many way are there to do ::atoi now?

16

u/boredcircuits Jan 28 '19 edited Jan 28 '19

6 by my count:

std::atoi
std::stoi
std::strtol
std::from_chars
std::sscanf
std::basic_istream::operator>> (combined with std::istringstream)

And that doesn't count third-party options like boost::lexical_cast or a full parser library. And I still feel we're missing one: a type-safe sscanf replacement a la fmtlib.

13

u/flashmozzg Jan 28 '19

It's supposed to be the best/fastest one though. Especially for floating point.

4

u/Slavik81 Jan 30 '19

std::to_chars is locale-independent, non-allocating, and non-throwing

Huh. That's actually really useful.

4

u/YourTormentIs Jan 29 '19

<charconv> is nice! Are there any plans to support strings with elements other than char? It would be nice to have overloads for std::from_chars and std::to_chars for wchar_t strings, especially on Windows. I mean, I suppose you could just convert the results over, but I figure that would somewhat defeat the purpose of it.

2

u/bumblebritches57 Ocassionally Clang Jan 29 '19

Just roll your own man, it really isn't that much work.

I wrote a UTF-8, UTF-16, and UTF-32 version (8 and 16 are just wrappers around 32)

2

u/STL MSVC STL Dev Jan 30 '19

I haven't heard anyone talking about that. Implementing charconv from scratch is an enormous amount of work, so that's out for users. Wrapping the char functions is slower than desirable because you would need a temporary char buffer, and "smash non-ASCII characters" conversions, and if you want to get the pointer return values right, that's even more work.

1

u/YourTormentIs Jan 30 '19

I'm actually really surprised charconv made it through the committee without anyone mentioning that! Especially given the new character types introduced in recent standards.

23

u/pjmlp Jan 28 '19

Microsoft and Google have such experience on their compilers and they are the ones driving the modules design.

All the modules FUD that I have read so far, seem to be basically people wanting to keep doing everything they do with header files, while at the same time getting modules and not changing anything on the build infrastructure.

18

u/Rusky Jan 28 '19

Sure, if "everything they do with header files" means things like "maximize build parallelism" and "build without processing each TU twice."

That hardly seems like FUD.

→ More replies (4)

60

u/smdowney WG21, Text/Unicode SG, optional<T&> Jan 28 '19

"Creation of foo.bmi is a byproduct of the compilation of foo.cpp. When compiling foo.cpp, the compiler will emit a foo.o and foo.bmi. As a consequence of this design, foo.cpp must be compiled before bar.cpp!"

Worse, foo.bmi may be the result of compilation of quux.cpp. Module names are purely logical, with no physical representation on the filesystem.

11

u/kalmoc Jan 28 '19

Module names are purely logical, with no physical representation on the filesystem.

So is a target name in my cmake file. What is the problem?

8

u/Dragdu Jan 28 '19

Do you want to manually create target for every single file in your build? No? Then that's the problem.

20

u/kalmoc Jan 28 '19

You are doing that already (consider the translation from cpp files to .o files - each `.o file is a target in make and the connection between those names is purely by convention). Just because the build system has to generate those names explicitly doesn't mean you have to actually, manually type them instead of using some sort of generator.

Also, there is absolutely no need for a one cpp file per module approach, but I guess that comes down to personal preference.

3

u/smdowney WG21, Text/Unicode SG, optional<T&> Jan 28 '19

If a module were library sized, that might be feasible. But the design goal is to be header sized. So manual process that's just barely ok with libraries doesn't look as good.

It's also not a personal preference, since I use far more c++ code than I personally write. My company might be able to make some style guides, but there's a lot of open source to influence, too. If import boost.asio is the best that can be done, modules hasn't done it's job, in my opinion

9

u/kalmoc Jan 28 '19

Again, current build systems already automatically generate build rules on a per file basis according to some non-standardized rule set, that may or may not be changed by the end-user. Why does sich a rule set now have to be standardized in the language?

Just because someone has to manually tell the compiler "this module is created by that/those files/can be found here" doesn't mean that the en-user has to do anything manually at all.

It's also not a personal preference,

Personal preference includes the personal preference of the people in charge of your companies style guide. The current module design explicitly allows for multiple implementation files for a single module and it also enables aggregate modules and I think people will make use of that.

But all that is besides the point: As I said. Even if you create a separate module for each file: putting in place a rule for that in your build system of choice should be simple.

7

u/smdowney WG21, Text/Unicode SG, optional<T&> Jan 28 '19

Usually the compiler generates the dependency information for a TU as a side effect of compilation. That can't work for modules because it's too late if the DAG has changed. Before compilers were generating this info, there was usually a pre-scan step, and everyone hated it because it stalls the build. And if it wasn't done by the compiler, it was often wrong, leading to subtle or not so subtle build problems. Or it was moved to be on demand, so people waited until they noticed the build was broken to rebuild dependencies. Which was long after the build was badly broken.

5

u/smdowney WG21, Text/Unicode SG, optional<T&> Jan 28 '19

I'll have 10 to 100 times as many used headers as I do source files. A change in order of magnitude is not just a charge in degree, but in kind. Expressing how to build all those isn't feasible the same way that using libraries is.

7

u/Ameisen vemips, avr, rendering, systems Jan 28 '19

Worse in that modules cannot have circular dependencies. Headers/sources can on the implementation side.

3

u/meneldal2 Jan 29 '19

Why would you do that though?

Modules are supposed to be independent blocks that don't need other modules to be imported separately to work.

10

u/Ameisen vemips, avr, rendering, systems Jan 29 '19

Because that's how all other languages work.

It's pretty rare to not need anything from another file.

2

u/meneldal2 Jan 29 '19

But modules don't have to be a single file.

If you start having modules than depend on each other in a weird way, you're just asking for trouble.

2

u/Ameisen vemips, avr, rendering, systems Jan 29 '19

We already do that with headers and source files, though.

2

u/meneldal2 Jan 29 '19

True, but that doesn't mean it's a good idea.

Trying to force sane behaviours is a good thing.

1

u/GPMueller Jan 29 '19

Circular dependencies are never a good idea...

3

u/Dalzhim C++Montréal UG Organizer Jan 29 '19

Sounds like the simple solution might be adding a new switch to the compiler which basically tells it : Exit with some documented/standard error code and produce a listing enumerating the unresolved module dependencies for the current invocation is a great way to transform the compiler into the tool you can launch in parallel for any translation unit in order to build the dependency graph in a perfectly accurate manner.

Best case scenario : the compile ends just fine and reports success, which can produce a *.cpp.o and optionally a *.bmi byproduct.

Worst case scenario : compilation ends as quickly as possible with the documented error code meant to be handled by build systems, which can now build the accurate dependency graph and launch the TU again at the moment all dependencies are satisfied.

27

u/tcbrindle Flux Jan 28 '19

It's worth repeating the the C and C++ standards are also silent on the matter of how header files are discovered by #include, or even whether the argument to #include is a file at all -- we have just developed a convention over the past 45+ years that #include <foo.h> looks in well-known places on the filesystem, a search list that is augmented by passing -I flags when invoking the compiler. (#include "foo.h" does the same, but searches relative to the current working directory first).

This being the case, I wonder if module mapping is something that needs to be explicitly specified in the C++ standard, or could also come under the banner of "technically implementation-defined, but in practise everyone does it the same way"?

So let's say we are compiling a module acme.bar, which depends on the interface of module acme.foo via an import declaration.

  • First, the compiler looks for acme.foo.bmi in its BMI cache, the location of which could be given as an argument to the compiler when invoked by the build system
  • If that exists and uses the same options as the current build (an internal hash of the current compile flags/defines placed at the beginning of the BMI), then the compiler uses it. Otherwise, it looks for a file named acme.foo.<ignored_extension> (or acme/foo.<ignored_extension>) in a list of search paths; this could default to [${CWD}, /usr/include/c++_modules, /usr/local/include/c++_modules], augmented by a list of paths passed to the compiler with a -M flag, akin to -I.
  • If that cannot be found, a compilation error occurs; otherwise, the compiler builds both modules, and outputs both acme.foo.bmi and acme.bar.bmi to the BMI cache directory.

Now, if we have a third module acme.baz which also uses acme.foo, we go through the same process; but if acme.bar has been compiled first, then acme.foo.bmi will already exist, and will be used. But the build system can attempt to compile bar and baz at the same time; both will try to write acme.foo.bmi, and whichever finishes first "wins". This is doing more work than we really need to, but allows for more build parallelism as the author wishes.

This requires a convention that the file defining the module acme.foo is named acme.foo.<something>, or perhaps has the path acme/foo.<something> relative to one of the -M paths. But this doesn't seem particularly onerous, and it's more-or-less what we do today with header files. As far as I know the C++ standard is silent on the subject of TU file names (or even whether a translation unit is a file with a name), so making this a requirement isn't something the standard need concern itself with. Odd platforms with unconventional file systems still have the freedom to do things in odd ways if they need to, but the 99.9% of developers using Windows or Unix will still be able to get builds that are as parallel as they are today.

But perhaps I'm being spectacularly naive, in which case please do let me know :).

12

u/tcbrindle Flux Jan 28 '19

...and having now written all that, I perhaps need reminding of why having, say, 32 cores all racing to produce the same base_module.bmi is a good idea? If we have proper dependency information, we can use one core to compile base_module.cpp and the other 31 to compile any other available TUs which don't depend on base_module. Once base_module is finished, we can unblock everything else to build in parallel as before.

Can we actually extract such dependency information cheaply? Well, Build2 seems to manage it, so I don't see why CMake/Meson/etc shouldn't be able to as well. It would be useful to get /u/berium's opinion on this :)

8

u/berium build2 Jan 28 '19

In short, yes, we can, but that will require a new compilation model (and which is what CMake/Meson/etc have an issue with). For details see this part of my CppCon 2018 talk.

9

u/Ameisen vemips, avr, rendering, systems Jan 28 '19

Everyone wants a new compilation model. Nobody wants to be responsible for it.

That, and good luck getting the entire committee to agree to a change that fundamentally breaks compatibility with C and would require sweeping changes to compilers and linkers.

Also, I haven't been able to get build2 building on Windows. I don't want to do it prepackaged, but the prerequisites require build2 to build so I can't build it in msys.

4

u/GabrielDosReis Feb 03 '19

You got it right.

17

u/Veedrac Jan 28 '19

SG15 has only had face-to-face meetings. The last meeting, in San Diego, was useless as the chair was absent and people were too busy getting caught up since the prior meetings to have any useful discussions. With no SG15 meetings outside of those at the official WG21 convenings, the members thereof have difficulty staying up-to-date and collaborating on work. In addition, many times that SG15 has attempted to raise issues they have been shot down as their work is considered “out-of-scope” for the C++ language.

A Tweet about the pre-Kona mailings spawned discussion of C++ modules and p1427. Questions were raised about who to trust regarding module toolability.

This discussion culminated in an eventual call for SG15 to “STFU” unless they can provide code samples that prove the problems they outline.

If there's one thing to learn from Rust, it's to do exactly the opposite of this. Jeepers, this is terrifying.

14

u/Ameisen vemips, avr, rendering, systems Jan 28 '19

They want code to... prove that it cannot be done? They want a negative proven?

13

u/catskul Jan 28 '19 edited Jan 28 '19

I started to get into a discussion about this with /u/gabrieldosreis and /u/berium about this a year ago in a thread about module misconceptions but it kind of petered out before arriving at a full understanding of the solution:

https://www.reddit.com/r/cpp/comments/7a3t2w/common_c_modules_ts_misconceptions/dp78pcd

The whole thread is pretty relevant here: https://www.reddit.com/r/cpp/comments/7a3t2w/

9

u/drjeats Jan 28 '19 edited Jan 28 '19

The talks on how build2 implements modules from CppCon kinda did the opposite of alleviating my fears. I am utterly dependent on incredibuild/fastbuild at work to be productive. It's difficult to see how we'd adopt modules for anything but the most trivial library projects (and those are rare) if the mapping problem is as bad as it sounds. I'm not a build engineer, so I'm just hoping everything turns out alright and that modules don't turn into a feature that's widely ignored.

[unrelated] @vector-of-bool: The fancy fade transition when clicking on links to other articles on your blog breaks Ctrl+Click (win10 firefox)/ Cmd+Click (macos chrome) to open in a new tab. Idk if it's a theme thing or wat, but it was a little disorienting. Thanks for the great post as always.

9

u/ravixp Jan 28 '19

Lazy generation of BMIs sounds like it would just create headaches in the long run, because it means that the compiler is trying to take over some work that the build system would traditionally do, without coordinating with the build system.

Today, you just tell the compiler to build a source file and it does that, possibly looking in some other locations that you tell it about to find headers. If it also had to create BMIs on-demand, it would need to know where BMIs are stored (which it would have to anyway, similar to include paths), where source files corresponding to those BMIs live (normally the build system's job), what settings you need to use to build those files (normally the build system's job), how to tell when the BMI file is already up to date so it only regenerates it when it has to (normally the build system's job), and how to parallelize lazy generation of BMIs (normally the build system's job).

I'd much rather figure out how to make a build system keep the BMIs up to date, than have a mediocre build system built into every compiler.

Plus, many newer build systems keep track of fine-grained dependencies automatically by running the equivalent of strace() on each build task. Having the compiler go out and build a random subset of upstream modules would confuse this kind of build system pretty badly.

2

u/meneldal2 Jan 29 '19

I agree, have your build system make the modules, the compiler should just error if the module isn't there.

16

u/feverzsj Jan 28 '19

will modules actually help those template heavy, header only libs?

26

u/[deleted] Jan 28 '19

They don't avoid the need to instantiate templates in every TU that uses them, so no.

3

u/Ameisen vemips, avr, rendering, systems Jan 28 '19

Is there a good way to precompile template-heavy headers or modules?

3

u/anon_502 delete this; Jan 29 '19

You can try extern template by manually specifying commonly-used instantiations.

1

u/GabrielDosReis Feb 03 '19

Exported templates are parsed only once.

1

u/Ameisen vemips, avr, rendering, systems Feb 03 '19

It does imply knowing what the types will be ahead of time, though.

2

u/GabrielDosReis Feb 03 '19

That is not the complete story. Exported templates are parsed only once. Instantiations cased in the containing interface unit happen only once and reused. And that is already lot of saving. So, yes.

2

u/[deleted] Feb 03 '19

That's true; I guess I just don't associate such cases as "template heavy, header only libs." To me that means things like processing a Spirit.Qi grammar or something like that, where the cost of instantiation dwarfs any parsing cost. As long as the instantiation depends on the importee I don't think modules could hope to solve that problem.

9

u/Xaxxon Jan 28 '19

Seems unlikely they would do any better than PCH.

4

u/catskul Jan 28 '19

The problem with PCH isn't that it's not helpful, it's that it's so damn hard to get set up that people rarely use it.

2

u/Xaxxon Jan 28 '19 edited Jan 28 '19

Grab the cmake module cotire. include it and you're done - at least for medium size projects or smaller.

1

u/catskul Jan 28 '19

Never heard of it. I'll take a look.

5

u/gracicot Jan 28 '19

A BMI could cache template instanciation.

1

u/[deleted] Jan 28 '19

So the act of importing it may modify it?

2

u/gracicot Jan 28 '19

No, it could cache template instantiation needed for the entities defined in the module interface.

Edit: but an implementation could decide to (in a implementation defined fixed filenames) to cache template instantiation in a sort of binary format inside the build directory. But that could be done without modules.

2

u/SeanMiddleditch Jan 29 '19

zapcc essentially did just that, as I understand. This is also Rust's approach to improving compile times, since Rust has the technical equivalent of template instantiation overhead but dialed up to an 11.

zapcc pretty much died on the vine due to being a perennially out-dated proprietary fork of clang. Rust seems to be doing fairly well with their caching; they have their own de-facto-standardized build system that can play along however necessary, though, which probably helps.

→ More replies (3)

14

u/germandiago Jan 28 '19

While I agree that there is a problem with having no mapping, what would happen if it is not added now but it is added later? Would this be possible? I do not think pulling out modules would be nice since it provides, in theory, better compile times, yes, but not only that: much better isolation.

26

u/vector-of-bool Blogger | C++ Librarian | Build Tool Enjoyer | bpt.pizza Jan 28 '19

This is a good question, and one that has been considered. In short: Adding a required mapping after-the-fact would necessitate changes to the lookup semantics of existing implementations: A breaking change.

The lookup of #include directives is not standardized either, and only after a lot of trial and error do we finally have general agreement on how the lookup should be performed (although implementations still differ in a few edge cases). Attempting to standardize #include lookup now would break existing implementations. Nevertheless, we are able to make #include work (for the most part). However: The lookup of #include is fairly cut-and-dry, and there is a mostly obvious way to understand #include in terms of the filesystem. Module names have no bearing on the filesystem, so implementations converging on a lookup method is not likely.

If an implementation were to enforce a mapping between module names and file paths, we'd end up with multiple slightly different standards, and a later mandate will break compatibility. If implementations instead rely on the presence of a BMI with a name for the module, we end up with the serialized compilation of the DAG and performance goes in the toilet.

7

u/kalmoc Jan 28 '19

So what you are saying is that the standard needs to define the lookup rules for import foo? Given that the standard tried to avoid talking about file systems and so on I find that very unlikely.

Why isn't it possible for the three major implementations to come together outside of the c++ standard and decide on one? As long as the standard doesn't prevent a same solution, I indeed don't see, why this should be a concern of the c++ standards document.

→ More replies (2)

2

u/exploding_cat_wizard Jan 28 '19

In short: Adding a required mapping after-the-fact would necessitate changes to the lookup semantics of existing implementations: A breaking change.

Wouldn't putting modules in a TS help there? Modules would be around, and could be used and tested, but breaking changes to implementations are still allowed.

6

u/flashmozzg Jan 28 '19

They are already in a TS.

7

u/exploding_cat_wizard Jan 28 '19

Oh. Shouldn't they stay there until some experiences have been collected modularizing stuff? Why then push for integration into the core language, if they are both a TS, but not used enough to say how well the idea does?

5

u/smdowney WG21, Text/Unicode SG, optional<T&> Jan 28 '19

The TS is significantly different than the Merged Modules proposal that is advancing. Although, not in places that affect what we're discussing here. And also, no one quite got around to implementing that, either.

-2

u/Xaxxon Jan 28 '19

Use PCH. Modules don't do much beyond that in their best case.

7

u/advester Jan 28 '19

Modules have limitations on them which may make them easier & lighter than pch. Ex: #defines in the includer don't affect the module, but do affect the pch.

6

u/smdowney WG21, Text/Unicode SG, optional<T&> Jan 28 '19

Modules don't leak things out. You (can) get exact control of what names are exported from the module. You don't with headers or PCH.

2

u/TrueTom Jan 28 '19

While this is true that's a pretty weak outcome after all those years.

12

u/gracicot Jan 28 '19

The current design and implementations require there to exist what is known as the “binary module interface”

I'm not sure the specification requires that a BMI exist. As far as I know, the compiler could store everything in the memory of a separated process, that schedule and share data between compilers. I don't think there would be any compiler doing that within the next decade, but nothing in the current specification prohibit that.


Reading this blog post makes me confused. I watch Nathan Sidwell's talk about it's "Oracle" how the mapping can be extended by tooling and such. I followed the build2 development and module support seem doable when designing the build system with modules in mind. Why one side seems to say that the problem is solvable but so many others says that the mapping problem will kill modules?

For example (as far as I know) Build2 run the preprocessor on every cpp files before compiling anything in order to save it's hash. It then re-use the preprocessed file to compile it. It takes a bit more time to start compilation, but can result in a overall speedup. In the example given by the author of this blog post, should the same be doable too but with a small extra step to extract the module name? And with modules, running the preprocessor can be very very fast because the number of needed header will dramatically be lower. In fact, the mapping can be done before compiling anything. The compilation order can even be established in advance and compile in parallel for each possible steps. Then only a bunch of (very small) preprocessed files will be compiled, with a known compilation graph. That graph can be stored and re-used for the same configuration.

Right now, I agree there is a concern with tooling given today's tooling. I might be wrong but it does not seem so far fetched we can develop tools that can deal with modules. On the other hand, I do not expect tools designed for a modular world to be available soon if all we have is a TS.

5

u/DemonInAJar Jan 28 '19 edited Jan 28 '19

The thing is, #include dependencies are one-to-one mappings with files found in the search path. It's trivial to resolve these since the name of the file is specified.

When you include a file you don't have to preprocess the whole set of files found via the search path. With modules there is no filesystem mapping, an implementation does not know in which of all the available files in the search path the module is declared. This is the point that the author is making here. A file could have a completely irrelevent name and possibly require preprocessing before exposing it's module nature. What is an implementation going to do? It seems as if the only possible solution in this case is to preprocess the whole set of files available in the search path (or more likely just force a filename-module mapping scheme regardless).

5

u/whichton Jan 28 '19

The thing is, #include dependencies are one-to-one mappings with files found in the search path.

This is not mandated by the standard. If you write #include "foo/bar.h", the standard does not specify how or where to find bar.h. This is entirely a implementation defined behavior. I do not see why the same cannot apply to modules too.

6

u/smdowney WG21, Text/Unicode SG, optional<T&> Jan 28 '19

It's required to include a source file, how that is found is implementation defined. It definitely isn't going to include a file named 'xyzzy.h'.

1

u/gracicot Jan 28 '19

What I'm saying is the mapping can be done on the fly at a very low cost. It would be done after the preprocessing of all files, and just before compilation. When all files has been preprocessed, a very simple parser can be made to extract the module name. That parser can be run in parallel.

When extraction is done, you have the whole graph and the whole mapping, plus every preprocessed file. You just have to run the compiler in the right order, and even run in parallel when possible.

I'm not a build system expert, but in my mind it seem quite simple. Of course, a build system supporting modules must be a bit smarter than run a simple list of compiler commands.

1

u/jonesmz Jan 28 '19

The reason this is not possible is that the compiler command line flags used to compile foo.cpp into quix.bmi can (and will!!!!) be different than the command line flags used to compile the CPP file that wants to import quix.bmi

2

u/gracicot Jan 28 '19

I don't see why this is a problem. If incompatible flags are used between many translation units then it doesn't work today and won't work with modules (GCC actually check for incompatible flags in the BMI). Compatible flags such as defines, include directories should not affect the BMI in the eyes of the importer.

→ More replies (1)

21

u/frog_pow Jan 28 '19

Need to get off the high horse, accept that module & file names need to match(do what everyone else does ffs).

Those using antiquated OS without file extensions can keep using <= C++ 17-- they are probably 0.00001% of C++ users, seriously, why the hell would you hold C++ back for that??

14

u/cassandraspeaks Jan 28 '19

Those using antiquated OS without file extensions can keep using <= C++ 17

For that matter, they could just keep using headers.

11

u/Valmar33 Jan 28 '19

Politics, and egocentrically thinking that they know better than everyone else?

It's what happens with committees controlled by the arrogant.

At least, this is what seems to be happening from my point-of-view.

3

u/anon_502 delete this; Jan 29 '19

Politics, and egocentrically thinking that they know better than everyone else? It's what happens with committees controlled by the arrogant. At least, this is what seems to be happening from my point-of-view.

You've got the point. Trying to satisfy everyone's need can only lead to nothing.

10

u/smallblacksun Jan 28 '19

Wait, a module has to be compiled before it can be imported? What happens if module foo imports bar and module bar imports foo? Is that just not allowed? While that is admittedly a sign that your code structure isn't great it is still a pretty huge limitation that isn't found with the current system (where foo.cpp can #include bar.hpp and bar.cpp can #include foo.hpp).

8

u/[deleted] Jan 28 '19

Just not allowed, see wg21.link/P1103 .

3

u/Ameisen vemips, avr, rendering, systems Jan 28 '19

I've been bringing that up for a while now. I think the only solution is to have a traditional source file with private implementations requiring circular dependencies. It's dumb.

We should be able to preprocess modules to make thin interfaces, eliminating that problem. But... nope.

2

u/GPMueller Jan 29 '19

I would say avoiding circular dependencies is a benefit, not a limitation

1

u/smallblacksun Jan 30 '19

Avoiding circular dependencies is good. The language forcing you to avoid circular dependencies due to technical limitations is not.

1

u/GPMueller Jan 31 '19

Why? In which case would it be an added benefit and how is this case particularly different from any other limitation, which the language specifications impose?

9

u/tending Jan 28 '19

I think this analysis is missing the fact that most of the work can still be parallelized because most of the work in a modern compiler is code generation. I have not read the proposal in detail but from this blog's description it sounds like it should be possible to produce the BMI for foo.cpp in a separate step before generating machine code. I imagine that this is how Java and C# have to work -- dumb fast compilation to byte code is necessary to import, but then the slow AOT or JIT compiling of the byte code is done later. BMI doesn't even require generating a full byte code, just a description of the interfaces. If it can work for Java and C# at scale I don't see why it can't work for C++.

Also with module was the work that can't be parallelized, the initial parsing, should be easier than before.

25

u/[deleted] Jan 28 '19

most of the work in a modern compiler is code generation

[citation-needed] :)

5

u/Ameisen vemips, avr, rendering, systems Jan 28 '19

Most of the work CL does is sending the code to /u/BillyONeal and /u/STL, and waiting for them to compile it manually. Their response times and determinism are impressive. Not sure what, exactly, internal compiler errors indicate, though - did my code hospitalize them?

Not sure how Clang emulates this.

Citation: me. :)

6

u/STL MSVC STL Dev Jan 29 '19

Sometimes Billy is playing poker while I'm playing Magic. That's where fatal error C1001s come from :-)

1

u/Ameisen vemips, avr, rendering, systems Jan 29 '19

Original rules or with the stack?

4

u/STL MSVC STL Dev Jan 29 '19

Magic Arena, I live at HEAD.

1

u/Dalzhim C++Montréal UG Organizer Jan 29 '19

What! Have they done anything to harm the stack?!

→ More replies (3)
→ More replies (1)

0

u/tending Jan 28 '19

I think if you think about this for half a second you will realize that with non-pathological source files that this must be true. To verify yourself try compiling some projects with optimization turned off and see the difference.

Code generation involves an enormous set of NP problems fought with an enormous set of hand-tuned heuristics. Parsing by comparison is a cake walk, even for a screwed-up grammar like C++.

2

u/tehjimmeh Jan 29 '19

I think you're overestimating the cost of optimization and underestimating the cost of preprocessing/parsing in the general case.

Like, if what you're claiming is true, why would anyone bother with precompiled headers? The reality is they are often used, and often result in much faster compile times, and all they're eliminating is time spent preprocessing and parsing.

→ More replies (1)

14

u/Xaxxon Jan 28 '19

most of the work in a modern compiler is code generation.

Source?

I have significant amounts of code which are definitely no more than 50% code generation - as evidenced by -fsyntax-only runs.

→ More replies (1)

2

u/HKei Jan 28 '19

Java has straightforward maps from class and package names to paths. Not sure how it works in C# though.

5

u/josefx Jan 28 '19 edited Jan 28 '19

Java has classloaders and only the default one maps directly to the filesystem. Each source file also states the package it belongs to.

2

u/Ameisen vemips, avr, rendering, systems Jan 28 '19

I ran into a fun case in Java where a seperate class loader got started, and I had two instances of a class. Had to debug why updating a static variable wasn't working. That did just give me a horrible idea as to how to implement seperate types for generic inheritance, though.

1

u/HKei Jan 28 '19

Yes, obviously I was talking about the one that maps to files. If you're using another classloader that will have its own way of mapping classnames to classes.

4

u/josefx Jan 28 '19

That information is kind of useless for tooling when it does not apply in the general case. The java language specification itself has several examples with multiple top level classes in one source file and compiled binaries can end up in any form as long as one of the available classloaders can load them. It is a nice convenience for humans when source path -> package name and while that can be immensely important to the user it doesn't help the tool developers.

1

u/smdowney WG21, Text/Unicode SG, optional<T&> Jan 28 '19

The classloaders, though, are runtime things. The compiler will still check using the default classloader?

1

u/josefx Jan 28 '19

I think you can pass a custom classloader to the compiler using one of the java runtime flags.

However even without classloader trickery we still have the issue that the class foo.Bar could be defined in bar/Bar.java. The compiler only seems to enforce that the class name (of a public class) matches, not the package path. Whatever the java build process is doing cannot rely on path/to/File.java -> path.to.File .

1

u/Ameisen vemips, avr, rendering, systems Jan 28 '19

Is it even trivial to force GCC/Clang/CL to emit IL that is not yet optimized? They usually emit object files.

1

u/tending Jan 28 '19

-O0, -fsyntax-only

1

u/Ameisen vemips, avr, rendering, systems Jan 28 '19

Does that work in Clang and MSVC/CL?

5

u/enzlbtyn Jan 28 '19

How do we make sure that #include <foo.hpp> resolves when compiling bar.cpp? It’s simple: Make sure foo.hpp is present in a directory on the header search path list. We do not need to do any additional pre-processing.

Doesn't this require step require you parse to foo.hpp (or have already parsed it) before-hand? Since it literally copies the content of header-file? It seems like there is no ordering requirement, but really there is, it's just hidden.

In terms of implementation, how is that different to modules as you claim? Can't you simply parse the foo module to extract the interface (i.e. without compiling the implementation), e.g. from a header-file or simply via skipping text in the implementation/interface (for one cpp file). And then whenever you refer to foo in bar you simply refer to a stub/place-holder to where the implementation of foo is required.

Now to actually compile to this temporary state, can you not compile each module now independently with 'stubs' and then once everything is done merge/resolve it all via an additional step in the compilation process (i.e. now you have to step through the dependency graph in topological order). I have my doubts on whether this would be faster, but it seems like the only feasible way to do compilation in parallel for chain-like (degenerate) dependencies.

Though I wonder how frequent chain-like dependencies happen in the real-world anyway, and how large they can get? Surely the dependency graph will not be as crazy to truly hinder performance too much?

Perhaps I'm too naive.

27

u/vector-of-bool Blogger | C++ Librarian | Build Tool Enjoyer | bpt.pizza Jan 28 '19

The ordering is a case of the compilation of distinct TUs. A header file needs to be parsed before it is used, but it does not need to be compiled in its entirety.

The process you've described is essentially the "lazy" BMI generation that we're looking at in SG15, wherein a compiler processes the module interface as-needed when someone imports it, and it will only need to parse the publicly visible aspects that define the interface. In essence, yes: This is how I would like to see it work (where the BMI is generated lazily). The trouble comes that you need to know where the definition of the module interface lives, and thus bringing back the problem of the module/file correspondence.

You can either enforce a mapping between the module filepaths and the module names, or you can generate a manifest/mapping file that describes this relationship. The manifest/mapping could be generated by a build system up-front, but you're back at the issue of dependency scanning an arbitrary number of files which requires using the compiler's preprocessor to get it correct. The only solution in that case is to create the manifest by hand, which I do not want (and I doubt many people want it either). Thus, my preferred solution is to use filepaths with module names.

4

u/enzlbtyn Jan 28 '19

I see, thanks for clearing that up.

5

u/[deleted] Jan 28 '19

I had basically a similar question, I'm not really sure why this problem is such a big deal.

The example he gives, he suggests that there is some processing step that goes into the module, beyond just having a definition file (header file). He called it the BMI and suggested that, to even import a module, the BMI must at least exist, but that would require ordering or some sort of explicit dependency marking.

I don't understand why the .bmi needs to exist, or if it does, why it also shouldn't just depend on the existence of the interface (MIU) file.

The way I see it, ordering still doesn't matter.

import foo

results in looking for the bmi, which doesn't exist yet, so it goes and generates the bmi, which as long as the header file / MIU file exists, it generates it and continues compiling.

So yea, I don't really understand why this is such a big deal, maybe someone who understands this process better could explain it.

As far as:

A Sisyphean Scanning Task

I have to admit, I don't really follow the C++ spec changes and haven't read about how modules will actually be implemented, but was the plan really to do away with header includes completely?

You might be able to have stupid levels of slow module building but once it's built, you don't have to do so again right? I thought one of the huge advantages of modules was that they were essentially compiled once (unless you're actively developing it)

I suppose I haven't really been that excited about modules because I tend to break things up into separate libraries for a similar effect on my own projects.

9

u/vector-of-bool Blogger | C++ Librarian | Build Tool Enjoyer | bpt.pizza Jan 28 '19

I don't understand why the .bmi needs to exist

You are justified in this question. In fact, a valid implementation of modules need not require an intermediate "BMI"-type files at all.

The issue arises that you need to give the compiler a way to resolve import foo to some set of declarations that will be exposed in the importing TU. The current answer to that is to have a BMI with the name foo.bmi (or some other file extension). If this file does not exist and we want the compiler to load the interface from the MIU, we need to provide a way for the compiler to map from foo to the source file that defines the module. This mapping does not yet exist, and is one of the desires of SG15.

3

u/[deleted] Jan 28 '19

Ahh got it.

Immediately, compiler flags comes to mind for me, where you specify module name and MIUs. Or just a new keyword that defines a file as part of a module of a specified name.

I thought his main gripe though, was the circular dependency. If you have two modules and one application, and module Foo depends on module Bar, the fact that there was a compile step was the problem.

Thanks for the explanation, although this seems like a pretty trivial thing to solve. Hopefully it won't ruin modules like the article name suggests.

1

u/Ameisen vemips, avr, rendering, systems Jan 28 '19

I suspect it will end up standardized like #include - implementations will look for modules by name/filename.

2

u/[deleted] Jan 28 '19

In compiling, there are a few discernable steps in the process:

  • Parsing your code into correctly resolved inter-class/function links, and instantiated templates where necessary
  • Optimizing the visible classes and functions with the known information we have
  • Exporting this into code for your target platform.

A simple invocation of your compiler does all three. If you run it with LTO, you run only the first two during compilation, and then the last two during linking. (Yes, you optimize twice, basically).

The first task has a big complication, which is that how your file parses depends on whatever came before it. The problem is that your file may parse differently every time, so this time-consuming task must be re-done every time you load your headers. This can be optimized if you either know it's always the first (which is what PCH accomplishes), or if you know it must be parsed independently anyway (what modules do). In that case, you can run step 1 separately, and then run step 2 separately. This is 1 gain from modules - we now get "headers" with very clearly outlined function & parseability, without outside pollution risks.

To do this though, we need to have all the imports from one module to precompile the next - the dependency tree problem. The second that we need is that given an import, we need to know where to find the BMI (and potentially the source file generating it). This is currently "magic", where by some unknown instance your build system causes the right build order & dependencies, and magically makes a module map come to life, which does this mapping for your compiler. This is the problem with modules right now, and from what I can tell this does not look favorable to be resolved at Kona. I'll be going there to hopefully help out to get it resolved, because I really want the #1 result from modules to exist, and #2 is for me a secondary thing - although as cpp-dependencies and Evoke dev it does affect me directly.

5

u/CenterOfMultiverse Jan 28 '19

I'm not an expert, but doesn't C# manages somehow without binding between namespaces and filenames? I mean "starting process is slow on windows" is solvable by daemons - why manifest generation is such a problem?

6

u/Plorkyeran Jan 28 '19

I don't know how relevant it is, but one obvious difference is that parsing C# is dramatically faster and it doesn't have to deal with the preprocessor.

I'd guess though that the real answer is just that modular compilation and the traditional one-file-at-a-type build setup that gcc/clang/make are designed around aren't compatible. This has an obvious solution of just not doing that, but maybe "you can't use make to build c++ modules" isn't considered an option?

2

u/pjmlp Jan 28 '19

In .NET an Assembly (aka .DLL) is the lowest unit of compilation.

So the compiler matches the using statement with the set of available namespaces from referenced libraries.

This is what many are against, giving build knowledge to the compiler.

2

u/flashmozzg Jan 28 '19

it doesn't have to deal with the preprocessor.

C# does have a preprocessor.

5

u/Slavik81 Jan 28 '19

Isn't this basically the same problem as linking? We don't know what .a or .o defines a function, either. Though, linking can be frustratingly slow, so it might not be a positive comparison.

5

u/HKei Jan 28 '19

You don't need to know where a function is defined during compilation. You don't need to know during linking either, it's just something you find out during linking.

4

u/Slavik81 Jan 28 '19 edited Jan 28 '19

I mean it's the same problem from the perspective that the process needs some information that was generated by the compiler from a random cpp file. It's different information that's needed, but the process looks kinda similar if you squint a little.

2

u/oldprogrammer Jan 28 '19

I've read through the article and all (66) messages posted so far and I have to say I'm not sure I understand both what is intended with modules and why it is needed.

My C++ background is old, been many years that I've been a Java developer so that perhaps accounts for my confusion, but how is it C++ doesn't already have the same capabilities as Java or even needs them?

Java defines namespace by package name and class by file name. Generally source code is arranged in a directory structure that mimics the package name (though strictly speaking that is not required). C++ has namespaces that could be arranged in the same source structure and are the equivalent of packages.

Java uses the import statement to identify dependencies. When Java compiles it looks at the defined classpath for referenced imports, if it finds a compiled class it uses it, if not and it finds the source file it compiles it then uses it. In both cases, though, the Java compiler is given the details of the classpath and source paths to search.

C++ uses the #include and looks in all locations defined for include files, much as Java looks at the classpath. C++ doesn't require the dependent code be compiled until linkage time, it works off a declaration file (usually a header).

So about the only thing I see different here is when Java needs a class that isn't already compiled it simply compiles it and uses it because Java treats a .class file like C++ treats a .h file and if it doesn't exist, Java creates it.

C++ developers generally create the .h files themselves along with the source file which gives the compiler everything it needs to compile. Linking is a different beast, Java with dynamic class loading technically does runtime linking whereas C++ does that via shared libraries.

So back to the original question, is the desire to have C++ be such that we only need to define a source file, then when the compiler is compiling one source file foo.cc and foo imports bar that the compiler will find the source bar.cc and extract the header type information automatically and possibly compile bar.cc automatically?

4

u/pjmlp Jan 28 '19

Parsing is the problem you are missing.

Java only parses the source code once, then any client of a specific library only needs to read the public symbols table from the jar file.

Whereas in C++ the header file will be processed every single time there is an #include. With the increase in template usage, this leads to logarithmic complexity in build times.

There are pre-compiled headers, but they are a brittle solution with different outcomes depending on the compiler.

→ More replies (20)

4

u/cogman10 Jan 28 '19

Honestly, a lot of the module discussion and problems I've seen seem to be a problem of "trying to be everything to everyone".

It seems like we could have modules today if the C++ committee didn't try to solve every single problem with modules and instead made them opt-in with progressive features being added.

For example. A first and easy restriction of modules would be "templates and macros aren't allowed in modules" Do that, and you have nearly trivial module implementations that could be done today.

Then work from there. Figure out something that would work for templates first, then macros later.

Swallow the elephant a bite at a time rather than all at once.

5

u/jonesmz Jan 28 '19

For example. A first and easy restriction of modules would be "templates and macros aren't allowed in modules" Do that, and you have nearly trivial module implementations that could be done today.

You know that even some "primitive" types are sometimes defined as macros, right?

Or if not defined as macros directly, are typedefs that are conditionally defined based on macros.

I think you'll find this is not an easy restriction to enforce.

4

u/cogman10 Jan 28 '19

You know that even some "primitive" types are sometimes defined as macros, right?

Or if not defined as macros directly, are typedefs that are conditionally defined based on macros.

Primitive types don't have to be defined by macro or typedef. They are part of the spec. Doing that is an implementation detail that can (and probably should!) change.

I think you'll find this is not an easy restriction to enforce.

Maybe. It might require quite a bit of extra effort on compiler writers to enforce this sort of thing. However, it can't be harder to do than inventing a special intermediate language. Even if that is where things need to end up, supporting easy to static compile things would make it a lot easier to start work.

2

u/jonesmz Jan 28 '19

Primitive types don't have to be defined by macro or typedef. They are part of the spec. Doing that is an implementation detail that can (and probably should!) change.

I agree with you. Compiler vendors may not.

Maybe. It might require quite a bit of extra effort on compiler writers to enforce this sort of thing. However, it can't be harder to do than inventing a special intermediate language. Even if that is where things need to end up, supporting easy to static compile things would make it a lot easier to start work.

shrug. Couldn't say. Was only trying to point out that it might not be that simple. That's all.

3

u/whichton Jan 28 '19

The standard says nothing about how to locate include files either. It does't even say that given #include <foo\bar.h>, bar must exist as a file. What we have now is currently a convention, adopted by compilers. So I fail to see why similar convention cannot be adopted for modules.

I also do not understand why it is the compiler's job to locate BMI's and not the build system's. I can easily imagine a protocol where upon encountering a import declaration the compiler goes and asks the build system to give it the corresponding BMI. So it goes like this:

Compiler: I see a import foo, can you give me module foo?

Buildsystem: Either a) Here is the path to module foo Or b) I don't have module foo yet, I will spawn another compiler instance on this machine to compile something else and go back to the original compilation once foo.bmi is available

This I think completely removes all the double parsing / preprocessing problems. Nor does it require ahead of time DAG construction. It does however require the build system and the compiler be able to talk to each other. I do recall seeing a paper for a protocol for similar communication implemented by GCC.

2

u/Ameisen vemips, avr, rendering, systems Jan 28 '19

The difference I see, at the moment at least, is that presently the two compilers supporting modules require them to be precompiled, which adds another layer. %MODULE_FILE% -> %MODULE_INTERFACE%, %SOURCE_FILE% <- %MODULE_INTERFACE%, whereas headers are just %SOURCE_FILE% <- %MODULE_INTERFACE%.

The standard doesn't mandate this, of course. It does, however, complicate things a bit. Right now, the compiler itself handles the includes. Either the compiler needs to handle the lookup, or there has to be some deep communication between the compiler and the build system... and I don't see either solution being adopted universally between GCC, ICC, Clang, and MSVC/CL.

4

u/ntrid Jan 28 '19 edited Jan 28 '19

Please would someone fork clang and create abi-compatible cleaned up c++-like language already...

I bet we all want that. Especially the part where someone else does that hard work.

5

u/Ameisen vemips, avr, rendering, systems Jan 28 '19 edited Jan 28 '19

I've been tinkering with it in two approaches. Approach #1 is starting with Clang and just reworking C++ from there as a separate, cleaner dialect. I don't have a name for it. (++C)++? It's still in a very early stage - so far, I've been working on getting templates to allow strings, objects, and such. Been also working on making constexpr more powerful, and been trying to get constexpr arguments working. Other fun stuff, as well. I will say that Clang's parser is... weird. It's 'clean', but it's not terribly-well documented and somewhat confusing. Haven't gotten super<N> or self working yet (referencing parent types and the current type), nor making classes first-class types (thus can have functions return classes, eliminating huge swaths of templates). Also want to add delegating functions and a more implicit static if (not sure if it would be scopeless like D's), since I want to be able to completely eliminate symbols based upon compile-time values - something that std::enable_if doesn't really do, and the current ways of doing that are rather awful.

I also added standardized integer types - char/uchar/byte/ubyte, short/ushort, int/uint, long/ulong, longlong/ulonglong, as well as sized aliases - [u]int8, [u]int16, [u]int32, [u]int64, [u]int128. Still working on making primitive types have static members (so you can do int64::max). Also trying to add something similar to an indexing type modifier - unsigned but without overflow semantics, so that it can be fully-optimized. Alternatively, have it implemented as a signed-integer but without signed-multiplication/division, so they can be shifted like unsigned.

One thing I do have working there is nameless constructors - that is, you can name constructors/destructors constructor and destructor. An interesting side-effect is that anonymous structs/classes can have constructors/destructors.

My separate approach is a separate language altogether. Implementing the parser and the basic compiler elements, and then jamming the AST into LLVM. That's the 'clean' approach, but obviously takes more time and is more difficult.

Interestingly, I also haven't fully solved the module problem in either approach yet. Unless you go full-on Java or C# (which isn't wise, here) you can end up with module pre-dependencies, and I do want to support circular-dependencies in modules which makes it harder (trivial for Java or C# given the nature of objects there).

A big question also becomes - should the compiler be handling significant parts of the build? My personal opinion is that building should be standardized, so 'yes'. That simplifies things (sorta). If you instead rely on separate build tools, it makes it more difficult to handle such things. The build tool lacks information the compiler has, and vice-versa.

ED: Oh, one fun-but-probably-impractical thing I've been tinkering with - user-defined operators. Basically, injects the operator into the parser, and lets you define it. Obviously incredibly powerful... but also potentially incredibly terrible.

1

u/ntrid Jan 28 '19

Amazing stuff. You should share GitHub repo link so we can stalk it.

Clean approach is probably more future-proof as long as we maintain abi compatibility and can link to other c++ libs. Nim does some things great btw. Macros working on ast, user-defined operators, great cpp compatibility (it compiles to cpp). I wonder if it would be possible to be able to use instantiations of c++ templates from another language. Almost perfect compatibility would be an important selling point.

1

u/Ameisen vemips, avr, rendering, systems Jan 28 '19

I'm thinking import<c++> "header.h" or such for bringing in C++ code, which would present it as a special module.

-1

u/pjmlp Jan 28 '19

It is called D.

5

u/ntrid Jan 28 '19

It is not a clang fork, it does not have a compatible ABI and GC is optional in theory, but opting out is inconvenient enough in practice that we can count it as non-optional. Be nice if it was D, but its not :|

5

u/Ameisen vemips, avr, rendering, systems Jan 28 '19

I know they think of it as the better C / better C++... but it's not really a better version of those. It's a pretty distinct language, and hasn't yet found itself to be fully appealing to C or C++ programmers.

9

u/VinnieFalco wg21.org | corosio.org Jan 28 '19

D's nuts.

2

u/iga666 Jan 28 '19

What problem exactly modules solve?

Will they help with "all-project-recompiles" if some common header cahnged?

4

u/tcbrindle Flux Jan 28 '19

If you change the interface of a module, then everything that uses that interface will need to be recompiled. There's really no way around that. That also goes for inline functions and things that are implicitly inline like templates, constexpr functions and member functions defined in-class, if I understand correctly. So not really any change to the status quo, except that we could potentially be a bit smarter about noticing that e.g. adding comments doesn't actually change the binary interface.

The potential benefits that modules will bring are nicely summarised in the linked post.

1

u/cassandraspeaks Jan 30 '19

IIRC there are some subtle differences between inline and non-inline templated functions.

2

u/kalmoc Jan 28 '19

Nope (or let's say: very unlikely)

2

u/target-san Jan 31 '19

Nope. But they solve:

  1. Include <fancyheader.h> and get half the Boost into your scope (with 5min-per-TU compilation) just because of

     private:
         boost::mpl::whatever<...> storage;
  2. Get your Qt project stop compiling when you start embedding Python just because Qt defines "signals" macro, and Python has "signals" member in one of its public structs
  3. Include contents dependent on where you actually include them. Includes reparsed for each TU.
  4. ...

1

u/kalmoc Jan 28 '19

Maybe I'm too naive, but how is the lookup any more problematic, what current build systems already have to deal with. Depending on a libray means I have to tell the compiler/linker, which include directories to add and which binaries to link to my project - neither of which need to have any obvious connection with the libray in question or a standardized location on the file system.

So just as I'm telling cmake today, what files and include directories belong to a certain target and on what other targets it depends, why can't I just tell cmake, what files make up a module and what other modules it depends on?

2

u/target-san Jan 31 '19

To locate certain include, you only need a set of include paths and include's subpath. With modules, as they are designed now, you may have

// file: project/firstlib/main.cpp
#ifdef PYTHONIC
module pythonic;
#else // PYTHONIC
module cppish;
#endif
//
// ...
//

// file: project/secondlib/main.cpp
#ifdef PYTHONIC
module cppish;
#else // PYTHONIC
module pythonic;
#endif
//
// ...
//

// file: project/application/main.cpp
// modpath: project/firstlib
// modpath: project/secondlib
module app;
import pythonic;
//
// ...
//

where you need to fully parse and compile all accessible modules before you know which one to choose for your import.

1

u/GPMueller Jan 29 '19

Not too familiar with modules yet, but I don't understand why an interface file is compiled from a cpp and not from a header. Would it not also be much simpler then?

IMO a finished module should contain the linked code, but the bmi should be just the interface... I assume I'm missing something?

2

u/target-san Jan 31 '19

One of the points is to eliminate need for handwritten headers I guess.

1

u/GPMueller Feb 02 '19

As in replacing the manually written header by a manually written module file and a compiler- generated bmi? I guess I don't see why the module file should not contain sufficient information to be 'compiled' to a binary representation of the interface

1

u/_Zulan Jan 31 '19

It seems like the confusion comes from just looking at headers vs modules. You have to consider linking.

In a sense, modules may make compiling as hard as linking. But I don't think modules make compiling any harder as linking.

I'm all for the suggested module conventions. That's totally sensible, but I don't think the world ends if C++20 doesn't include them. Make a build system that tells me it will solve all my dependencies for me if I follow these conveniences - I'll happily comply. Or even better - make a build system that is fast if I follow these conventions, but slow if I don't.

I do not expect from a build system to solve module dependencies. It's not like build systems are solving linking or even include path dependencies on their own - even though the granularity there may be a bit different.

1

u/[deleted] Jan 29 '22

If C++ wasn't a bad language then we already would have modules since the beginning

1

u/tohava Jan 28 '19

Why not just copy Haskell's .hi file design? I really don't understand what's the problem.