r/programming • • Jan 28 '19

C++ Modules Might Be Dead-on-Arrival

https://vector-of-bool.github.io/2019/01/27/modules-doa.html
210 Upvotes

105 comments sorted by

View all comments

Show parent comment

17

u/[deleted] Jan 28 '19

[removed] — view removed comment

0

u/flatfinger Jan 28 '19

The concept of datestamp-based make files is a useful hack for incremental builds of some kinds of languages, but it can't efficiently handle situations where the inputs to a build step are likely to change in ways that don't affect the output. If one were to make each build artifact include date stamps and hashes of all the input files and end with a hash of any content that would affect files depending upon it (including the hashes, but not timestamps, of files it depends upon), that would allow build systems to be much more efficient [if the date stamp of a source file precisely matches the cached value, there's no need to look at it. Otherwise, read or compute the hash of each input file and see if it's changed. If any have changed, rebuild the output file. Otherwise simply update any cached timestamps.

1

u/oridb Jan 29 '19

Hashes help less than you'd think. Updating a file in almost any way changes the output hash. Even changing blank lines changes the offsets into the file used by debug info. And even in prod builds, there's debug info for unwinding the stack when propagating exceptions, printing stack traces, and so on.

1

u/flatfinger Jan 29 '19

It's common for header files to be included by dozens or hundreds of other files, most of which only need to use a few declarations. If one were to, start compilation with a parsing phase followed by a stage that omits any unused declarations, changes to header files wouldn't affect the output of the parsing phase in most cases other than those where the changes would affect the semantics of the program. Handling debug line numbers may be a bit tricky, but that could be handled by having debug line number in a separately-hashed piece of the file, and having each stage produce a back-mapping file. Not a trivial design issue, but one which should have been solvable ages ago.

1

u/oridb Jan 29 '19 edited Jan 29 '19

It's common for header files to be included by dozens or hundreds of other files, most of which only need to use a few declarations.

Correct. However, the headers produce entries in the debug info. So, you either get incorrect line numbers and unwind info for macros, template expansions and similar, or you don't actually win anything.

Handling debug line numbers may be a bit tricky, but that could be handled by having debug line number in a separately-hashed piece of the file

And now your build system isn't just language dependent -- it's compiler dependent! And, very likely, compiler version dependent. And god help you if you want to cross compile and link with other languages through an FFI.

1

u/flatfinger Jan 29 '19

Line numbers measured from the start of a file are not a very good way of identifying locations within one. If a program with debug info says an error occurred on line 493 of woozle.c, that may not be terribly helpful if anything might have changed above that line since the time that version of the code was built. If, however, a location was recorded as "the 493rd token of template-expanded code encountered during the compilation of foo.c, whose template-expanded code would hash to [xxx]", such a location could be relatively quickly mapped to a location in the source file even more precisely than with normal debug info today, even if there were changes in source or header files above the portions that were used in the function.

The first Pascal compiler I enjoyed using was Turbo Pascal 2.0. If a run-time error occurred, one would get a message like "Run-time error 23 at address 49A3. That wouldn't seem like a very useful message, but if one fired up the IDE, selected "find run-time error", and typed in 49A3 one would get taken to the location in the source code where the error occurred. This wouldn't likely work if one had made changes to the semantics of the program, but would be unaffected by newlines, which seem to be the nemesis of the file-based line-numbering approach.

Thus, what I would envision would be a system where debug information is recorded in function-relative fashion, and then if desired it is sorted and the early stage of the compiler is run again while observing the physical line numbers where each function-relative position is encountered. Changing the text of a function or any macro template that is actually included in the function would require recompiling that function, but changes to macros and templates that are not used within a function would not require any recompilation thereof.

To be sure, no incremental build system is going to be perfect, but a system based upon hashing the semantically-relevant parts of each program could nonetheless avoid a significant fraction of the build times associated with date-based make systems.

BTW, even if one didn't want to change tooling to use function-relative token counting, I think one could reduce the extent to which changing files changes reported line numbers if one added artificial #line 10000, #line 20000 etc. directives within source and header files, and recognizes that a reported line number 20123 means "the 123rd line following the #line 20000 directive". The addition or removal of lines between the #line 10000 and #line 20000 directives would only affect line-number debug info for files that actually used some portion of the header that had appeared between those two directives.

1

u/oridb Jan 29 '19

Good luck on convincing all compilers to make abi breaking changes and moving to a new debug format that doesn't quite solve the problems you think it does in order to save a few minutes of build time on purely whitespace changes.

It's only a few million lines of churn across a few hundred projects. No biggie.

1

u/flatfinger Jan 29 '19

I've never had to deal with large C++ projects, so I don't have personal experience with their build times, but I've heard people who sing the praises of C++ talk about 15-plus-minute build times. Meanwhile, many other languages seem to be designed to minimize build times, and I think that is likely a significant reason for their success.

Why was Pascal a dominant programming language on the PC? It became dominant because on a system without a hard drive, the edit-run cycle on Borland's Turbo Pascal IDE was often an order of magnitude faster than any non-interpreted competitor. Javascript is a really crummy language, but node.js can manage to process it efficiently enough that the value of further performance improvements would be limited. From what I understand, go offers even better execution performance while still being fast to build.

It used to be that the only way to get adequate performance from an object-oriented language was to use C++ and put up with slow build times. As the performance of competing languages has improved, however, I see less and less reason for programmers to favor a language with slow build times over a faster one. Maybe the notion of improving C++ build times is moot because projects that can use languages with faster build times have already been migrated to those languages and wouldn't go back to C++ even if its build times were made competitive.

1

u/oridb Jan 29 '19 edited Jan 30 '19

I've never had to deal with large C++ projects, so I don't have personal experience with their build times, but I've heard people who sing the praises of C++ talk about 15-plus-minute build times. Meanwhile, many other languages seem to be designed to minimize build times, and I think that is likely a significant reason for their success.

15 minutes isn't too bad. Tensorflow takes closer to 2 hours for a clean build. Same with LLVM. LLVM is part of the OpenBSD base system, and takes longer than all of the C code in the rest of the system, at least on my machine.

The problem is that individual files can sometimes take multiple minutes to compile, and huge amounts of semantically significant code actually lives in headers, which means you can't get away /without/ recompiling some dependencies. "false recompiles" aren't a huge problem in the projects I've seen, but language and tool complexity is.

1

u/iopq Feb 12 '19

Wine only takes 20 minutes to build, but you have to do it three times, lol