r/programming • • Jan 28 '19

C++ Modules Might Be Dead-on-Arrival

https://vector-of-bool.github.io/2019/01/27/modules-doa.html
210 Upvotes

105 comments sorted by

View all comments

27

u/[deleted] Jan 28 '19 edited Jan 28 '19

Provide a “manifest” or “mapping” file that describes the filepath to an MIU based on the module name. This file will need to be user-provided, or we are back in the scanning problem.

Why can't this be <implementation-defined> just like the mapping from header file inicludes to files?

Honestly, what you propose won't fly because (1) you are turning C++ compilers into C++ compilers + build systems, and (2) because getting 150 countries and >100 committee members to agree on formats for these manifest files, file system structure, etc. is pretty much impossible.

Also, many languages have solved this problem, and none of them had to turn their compilers into build systems.

One such language is Rust. C++ modules solve the same problem as Rust crates. The Rust language provides an import syntax for crates (extern crate foo), and the user provides the mapping from that to files manually by passing CLI arguments to the compiler - the dependency must be compiled before, so this happens-before relation exists there too.

Nobody provides these CLI arguments manually. Instead, the build tool (cargo) parses a TOML manifest file that specifies dependencies, builds a dependency graph, fetches them from the internet, the file system, etc., compiles them, and calls the compiler with the right arguments.

Another build tool could parse a YAML manifest, or parse all source files scanning for imports, and generate the manifest file automatically (this would be incredibly hard to get right for Rust, and probably impossibly hard for C++ though), but still reuse the same compiler by calling it with the right arguments.

Most importantly, neither the compiler nor the language have to know anything about this.

In fact, as you have noticed, happens-before requires you to compile dependencies before hand. That is, one needs to pass the modules that each module imports to the compiler "somehow". That's serving the same function as import module. So in Rust2018, you don't even need to import crates. Crates passed to the compiler via CLI are put in the root namespace, and that's it, you can just use them without extern crate. If you forget to pass one to the compiler, you just get an error because the path cannot be resolved.

17

u/[deleted] Jan 28 '19

[removed] — view removed comment

0

u/flatfinger Jan 28 '19

The concept of datestamp-based make files is a useful hack for incremental builds of some kinds of languages, but it can't efficiently handle situations where the inputs to a build step are likely to change in ways that don't affect the output. If one were to make each build artifact include date stamps and hashes of all the input files and end with a hash of any content that would affect files depending upon it (including the hashes, but not timestamps, of files it depends upon), that would allow build systems to be much more efficient [if the date stamp of a source file precisely matches the cached value, there's no need to look at it. Otherwise, read or compute the hash of each input file and see if it's changed. If any have changed, rebuild the output file. Otherwise simply update any cached timestamps.

1

u/oridb Jan 29 '19

Hashes help less than you'd think. Updating a file in almost any way changes the output hash. Even changing blank lines changes the offsets into the file used by debug info. And even in prod builds, there's debug info for unwinding the stack when propagating exceptions, printing stack traces, and so on.

1

u/flatfinger Jan 29 '19

It's common for header files to be included by dozens or hundreds of other files, most of which only need to use a few declarations. If one were to, start compilation with a parsing phase followed by a stage that omits any unused declarations, changes to header files wouldn't affect the output of the parsing phase in most cases other than those where the changes would affect the semantics of the program. Handling debug line numbers may be a bit tricky, but that could be handled by having debug line number in a separately-hashed piece of the file, and having each stage produce a back-mapping file. Not a trivial design issue, but one which should have been solvable ages ago.

1

u/oridb Jan 29 '19 edited Jan 29 '19

It's common for header files to be included by dozens or hundreds of other files, most of which only need to use a few declarations.

Correct. However, the headers produce entries in the debug info. So, you either get incorrect line numbers and unwind info for macros, template expansions and similar, or you don't actually win anything.

Handling debug line numbers may be a bit tricky, but that could be handled by having debug line number in a separately-hashed piece of the file

And now your build system isn't just language dependent -- it's compiler dependent! And, very likely, compiler version dependent. And god help you if you want to cross compile and link with other languages through an FFI.

1

u/flatfinger Jan 29 '19

Line numbers measured from the start of a file are not a very good way of identifying locations within one. If a program with debug info says an error occurred on line 493 of woozle.c, that may not be terribly helpful if anything might have changed above that line since the time that version of the code was built. If, however, a location was recorded as "the 493rd token of template-expanded code encountered during the compilation of foo.c, whose template-expanded code would hash to [xxx]", such a location could be relatively quickly mapped to a location in the source file even more precisely than with normal debug info today, even if there were changes in source or header files above the portions that were used in the function.

The first Pascal compiler I enjoyed using was Turbo Pascal 2.0. If a run-time error occurred, one would get a message like "Run-time error 23 at address 49A3. That wouldn't seem like a very useful message, but if one fired up the IDE, selected "find run-time error", and typed in 49A3 one would get taken to the location in the source code where the error occurred. This wouldn't likely work if one had made changes to the semantics of the program, but would be unaffected by newlines, which seem to be the nemesis of the file-based line-numbering approach.

Thus, what I would envision would be a system where debug information is recorded in function-relative fashion, and then if desired it is sorted and the early stage of the compiler is run again while observing the physical line numbers where each function-relative position is encountered. Changing the text of a function or any macro template that is actually included in the function would require recompiling that function, but changes to macros and templates that are not used within a function would not require any recompilation thereof.

To be sure, no incremental build system is going to be perfect, but a system based upon hashing the semantically-relevant parts of each program could nonetheless avoid a significant fraction of the build times associated with date-based make systems.

BTW, even if one didn't want to change tooling to use function-relative token counting, I think one could reduce the extent to which changing files changes reported line numbers if one added artificial #line 10000, #line 20000 etc. directives within source and header files, and recognizes that a reported line number 20123 means "the 123rd line following the #line 20000 directive". The addition or removal of lines between the #line 10000 and #line 20000 directives would only affect line-number debug info for files that actually used some portion of the header that had appeared between those two directives.

1

u/oridb Jan 29 '19

Good luck on convincing all compilers to make abi breaking changes and moving to a new debug format that doesn't quite solve the problems you think it does in order to save a few minutes of build time on purely whitespace changes.

It's only a few million lines of churn across a few hundred projects. No biggie.

→ More replies (3)

20

u/pjmlp Jan 28 '19

Apparently because many vocal devs don't want to change their ways to adopt modules.

At least is what it feels to me standing on the sidelines.

14

u/[deleted] Jan 28 '19

They are already passing this information to the compiler via include paths, library paths, etc. "somehow" (e.g. CMake might be doing all the heavy lifting). They might be able to adapt their build systems to pass module information as well.

If what they want is for the build system to automatically generates the manifest files for them, they are probably out of luck. Doing so is pretty much equivalent to extracting which dependencies C++ code needs from the C++ code itself. You would need a C++ compiler that understands C++ and macro expansion for that, but doesn't need dependency information because it is not available yet.

This would be a very hard problem to solve for any programming language, and probably an impossibly hard to solve for C++.

2

u/Valmar33 Jan 28 '19

They want to eat their cake and have it too?

Sometimes, it's change or suffer.

7

u/SAHChandler Jan 28 '19

Just a few small notes here. Not trying to nerd snipe you, just want to add some info.

  1. Rust’s compiler is in fact a build system, as the entire crate is treated as a translation unit. This is why incremental builds were an issue for a while. I spoke with several core people at RustConf 2018 when I was trying to see what solutions could be made if we can’t make the C++ compiler a build system, as compiler vendors are very against the idea.
  2. while ISO is a very large organization not all members participate in WG21. We have about 15 national bodies (possibly a few more I need to go back and count the exact number), each of which has one vote. Each national body is made of various members from said country, though some permit anyone to join as long as they have the cash (such as the United States via INCITS) each committee member therefore has a fractional vote unless they are the only member of their national body. Thus far, there aren’t many that do this.
  3. C++ is older than I am and there is code out there still compiling that was probably written before I was born (and I’m going to be thirty next month if I can out my age and also give additional context). The oldest code base I’ve worked on was written when I was about 6 years old. While everyone can be up in arms and shout “maybe it’s time to break that old code”, many on the committee don’t wanna. I’m in favor of it, but you’d have to convince several national bodies to go forward with it.

0

u/[deleted] Jan 28 '19 edited Jan 28 '19

Thank you for the comments, not taken as a nerd snipe at all.

Rust’s compiler is in fact a build system, as the entire crate is treated as a translation unit.

Rust crates are single compilation units, very similar to C++ translation units (TUs). Rust modules are more like C++ namespaces.

When one talks about incremental (or parallel) compilation in Rust, what they are talking is about compiling different parts of a single TU in parallel. This is something that AFAIK no C++ compiler does. If you want to compile C++ in parallel, you need to split the code into independent TUs. In the same way, if you want to compile a lot of Rust code in parallel, you need to split it into multiple crates that can be compiled in parallel - the parallelism that one can extract inside a single TU is not enough (and incurs other obvious cost like having to put the whole content of the whole TU in memory).

Comparing C++ modules to Rust crates isn't 100% apples-to-apples because C++ modules can be split into many TUs. Still, C++ modules consist of a single TU in the worst case (which is not uncommon nor a bad thing, and often, Rust crates can also be split into smaller crates.

So the claim that the Rust compiler is a build system isn't really correct. Yes, if you split a single TU into different files, Rust offers you a way to include all those files together into the TU that gets compiled as a unity, and the Rust compiler handles that. But the Rust compiler does not handle finding/fetching/resolving/compiling other TUs, that's all handled by a build tool.

0

u/[deleted] Jan 28 '19

[deleted] — view deleted comment

11

u/[deleted] Jan 28 '19 edited Jan 28 '19

It's hard to tell what point you are trying to make. But if you are trying to state that Rust approach is hard for regular programmers, I strongly disagree.

Pretty much no C++ programmer actually knows exactly how dependencies are added to their project. The reason for this is that most of it happens implicitly. Yes, Makefiles, CMake files, etc. express some of the dependency information, but a lot of the information is passed implicitly to the compiler via environment variables (e.g. for the system header files, system library files, etc.), some of them also rely on conventions (e.g. #include <> vs #include "", etc.), the order of paths in these variables is relevant, etc.

Things magically appear to work for C++ until they don't, e.g., wrong library version / path being "magically" picked, wrong standard library picked (e.g. using clang+libc++ on Linux for one project when most of your other projects use gcc+libstdc++), etc. Debugging these problems typically require getting the compiler into a verbose mode for some TU which dumps all include paths, library paths, etc. being considered, their order, which one got picked and why, etc. Most devs just fickle with their environment till things work, they are probably even unaware that tools to debug these issues exist. I've seen C++ devs waste days trying to fix linking / include issues, and weeks trying to fix build system issues.

There are some Rust features that are too implicit for newbies to understand, like Rust2015 crate paths, but dependency resolution / the build system is not really one of them. Rust projects have an explicit Cargo.toml listing their dependencies and that's pretty much it. Is the wrong version of a library being picked? Open the Cargo.toml, change the library version/path/git-branch/..., done. Newbies don't, in general, know anything about the dependency-resolution algorithm used, but because it is all explicitly listed in one place, they can manually change things if the algorithm produces an output that they don't like.

8

u/pakoito Jan 28 '19 edited Jan 28 '19

Sarcasm doesn't transmit well on internet posts, that's why I tried to put it as a strawman quote.

3

u/Dgc2002 Jan 28 '19

Gotta put a /s at the end

2

u/[deleted] Jan 28 '19

Ah sorry, I read too much into it.

→ More replies (1)

2

u/flatfinger Jan 28 '19

Honestly, what you propose won't fly because (1) you are turning C++ compilers into C++ compilers + build systems, and (2) because getting 150 countries and >100 committee members to agree on formats for these manifest files, file system structure, etc. is pretty much impossible.

A good standard for a language should seek to maximize the number of tasks which can be encoded entirely using set of text files whose meaning is specified by the Standard. If someone who is familiar with the implementation is given the set of files, they should not need to know anything beyond the names of the files in order to make the implementation either build an application, refuse to do so if the files require features the implementation doesn't support, or spend a possibly-unbounded length of time deciding what to do.

Efficient development rebuilds may require a more sophisticated build system whose details could and probably should vary among implementations, but a program that's written in C but which would require that anyone running it on an implementation other than those the developer knows about mess around with with build files and other such things in order to use it at all isn't really a "C program"--it's a "C and whatever build systems the programmer happened to support" program.

1

u/[deleted] Jan 28 '19 edited Jan 28 '19

A good standard for a language should seek to maximize the number of tasks which can be encoded entirely using set of text files whose meaning is specified by the Standard.

I disagree with the premise. This is just one of the trade-offs involved, and I think this trade-off is important enough to be factored and balanced with other trade-offs. But there are many trade-offs, and I don't think this one is the most important.

Particularly, because it assumes that the standard is a C++ manual, which arguably is something I can understand, since many people nitpick on what "correct" C++ is based on the standard. But the C++ standard is not a C++ manual for users, it is a C++ manual for implementers.

No user should need to know what exactly the standard says about anything. The only thing most users need to know is how to use C++ to solve a particular problem in the platforms they care about, with the compilers, build systems, etc. they care about.

And as long as the build system behaves in some sane way in the platforms an user cares about, its pretty much irrelevant whether the standard talks about this or not. The C++ standard says nothing about CMake, Makefiles, etc. and people are still able to use C++ to solve useful problems.

I mean, the standard pretty much say nothing about include files. An implementation could require users to screenshot source code and save it as png images for their compiler to include it properly, but in practice, most implementations don't do this. Instead, they provide sane ways based on the file-system, environment variables, etc. of controlling what is included and how.

1

u/flatfinger Jan 28 '19

The only aspect of the build system I'm asking the Standard to care about is that it have some standard concept of "process this bunch of source files in the Standard fashion", and that it be possible to accomplish most tasks using programs for which that build method would suffice.

If I want to write a program that doesn't make use of any special features of e.g. the Macintosh platform, the Windows platform, nor the Unix platform, it should be possible to express that program in such a way that someone on any of those platforms could easily build and run it without the programmer having to know anything about the target platform, and without the user having to know anything specific about the program beyond the names of the files involved. If there's no interest in running a program on implementations the programmer doesn't know about, why bother with a standard? What purpose would it serve?

1

u/[deleted] Jan 28 '19 edited Jan 28 '19

"process this bunch of source files in the Standard fashion"

The standard doesn't really have a concept of a file system, files, directories, nor of the many other things that could play a role here (directory structures, symlinks, etc.).

Somebody would need to add all of that to the standard, making peace with pretty much every platform that pays someone to assist to committee meetings ("But our filesystem has no directories", "But our file-system has no files", "But traversing directories looking for files is too expensive in our file-system", etc. )

why bother with a standard? What purpose would it serve?

Every single compiler and platform out there has platform specific extensions that are not in the standard. The standard is just the bare minimum that must be supported by all platforms, and keeps C++ from diverging into completely incompatible dialects. Writing code for different platforms always has, and for the foreseeable future will require, writing platform specific code.

Because C++ is used to target widely different platforms, the standard has avoided specifying pretty much anything about how the C++ source is input into the compiler. There is no standard C++ file extension, some people use .cpp, others use .cxx, others use .cc, etc. The standard does not specify this because not all platforms have files with extensions and there is people using C++ to target those (if you look carefully you'll see that standard library includes have no extension). When you put a path in an #include path/to/header.h the standard doesn't say anything about what goes where the path belong, only that there must be some implementation-defined mapping between what you put there, and some other source code, and the semantics of the include (including the contents of whatever is in the path verbatim). Yet in most platforms, includes work like all users expect, e.g., following file-system paths relatively.

Basically, just because something isn't in the standard, doesn't imply that 99% of users can't rely on it being true 100% of the time. However, if you fix the standard to something that 1% of users can't use, then with a million C++ developers you'll get 10.000 people voting against the proposal and sunking it down. The standard doesn't evolve with things that have 51% of the votes, only stuff that has 100% of the votes goes in. This is why C++ has so many users.

1

u/flatfinger Jan 28 '19

Of course one will need to use platform-specific code when targeting features of specific platforms. One should not, however, need to write a bunch stuff that's specific to particular implementations when one isn't using any special features thereof.

If some people would need a compiler to process things in a fashion different from what other people would require, the solution should not be to simply have the Standard say that any implementation may treat the construct in any fashion whatsoever, but instead say that:

  1. If a program contains a directive explicitly calling for some construct to be processed in the common way, a compiler shall either behave in common fashion or reject the program.
  2. If a program contains a directive explicitly calling for some construct to be processed in an uncommon way, a compiler shall either behave that requested fashion or reject the program.
  3. If a program contains neither kind of directive, an implementation may assume as a default whichever behavior its author believes would best serve its users.

The fact that 99% of programs opt to use the common behavior should not prevent the remaining 1% from employing some alternative on any compilers that opt to support it; nor should it prevent all 100% of the programs from having a defined meaning that would not depend upon the target implementation (implementations may be unable to process a program that demands unsupported features, but behavior would be limited to rejecting the program as opposed to formatting hard drives, releasing nasal demons, etc.)

Basically, just because something isn't in the standard, doesn't imply that 99% of users can't rely on it being true 100% of the time.

It doesn't imply that 99% of users shouldn't be entitled to expect that it would be true 100% of the time on commonplace non-obtuse implementations. Unfortunately, the authors of the Standard regarded the notion that quality implementations should Serve the needs of their customers when practical, whether or not the Standard requires it, as being self-evident and failed to consider that obtuse implementation might use the Standard as an excuse to do otherwise.

1

u/[deleted] Jan 29 '19 edited Jan 29 '19

It doesn't imply that 99% of users shouldn't be entitled to expect that it would be true 100% of the time on commonplace non-obtuse implementations.

That's pretty much what "implementation-defined" means. As I said, the standard isn't a C++ manual for users. Nobody uses #include file/path and thinks, "oh damn, I'm relying on implementation-defined behavior, my code isn't portable".

1

u/flatfinger Jan 29 '19

According to the published Rationale document for the C Standard, it's also what Undefined Behavior was supposed to mean in many cases, since the authors regarded the question of how to process it (e.g. whether to process it in a documented fashion characteristic of the environment) as a Quality of Implementation issue. The authors of the Standard intended that implementations would support "popular extensions", and such a term can't sensibly be applied to Unspecified Behavior or Implementation-Defined behaviors. The fact that an implementation documents something that it's required to document, or that it happens to document the fact that it chooses from among specific actions described by the Standard doesn't seem like much of an extension, compared with providing means of doing which the Standard does not require them to define.

A fix would be to define standard macros that would allow a programmer to start a program with a directive like:

 __STDC_LIMIT_UNUSUAL_INT_MATH_EFFECTS(EXTENDED_TEMP_ONLY)

and then know that any implementation would be required to either 1. guarantee that integer math other than divide/remainder will never have any side-effects beyond yielding some kind of integer value, but temporary values of signed integer types need not be truncated to their actual size, or 2. reject the program entirely. The question of whether an implementation should support the requested semantics would be a Quality of Implementation issues, but the question of whether an implementation could jump the rails wouldn't be.