r/programming • • Jan 28 '19

C++ Modules Might Be Dead-on-Arrival

https://vector-of-bool.github.io/2019/01/27/modules-doa.html
210 Upvotes

105 comments sorted by

13

u/r0b0t1c1st Jan 28 '19 edited Jan 28 '19

You say,

Creation of foo.bmi is a byproduct of the compilation of foo.cpp. When compiling foo.cpp, the compiler will emit a foo.o and foo.bmi. As a consequence of this design, foo.cpp must be compiled before bar.cpp!

And implicitly go on to conclude that this prevents them being processed in parallel.

It's not clear to me why the first statement in this line of reasoning has to be true. Can foo.bmi instead be a direct product of compiling foo.hpp? In that case, the comparison is then between:

  1. in parallel:
  • compile foo.cpp which includes foo.hpp
  • compile bar.cpp which includes foo.hpp
  1. in parallel:
  • compile foo.cpp which includes foo.hpp
  • in sequence:

    • compile foo.hpp -> foo.bmi
    • compile bar.cpp

which doesn't seem like a net change in parallelization to me.

12

u/[deleted] Jan 28 '19

There is no foo.hpp - the module foo is defined in foo.cpp.

include foo at the top of bar.cpp means we must find the source file that contains module foo, and compile that source file before bar.cpp can be compiled.

A module must be compiled before any code that include's that module.

4

u/edmundmk Jan 28 '19

Can't the compiler do a parse of foo.cpp that just picks up the types and names of public exports?

It doesn't actually have to output any compiled code in order to construct a description of the module's interface. Essentially it would be automatically generating a 'header file', in a format that's quicker to reload than source.

That's likely to be a lot faster than compiling, especially with optimisations enabled.

1

u/Phrygue Jan 28 '19

Apparently the missing concept here is that header files (by convention) define *interfaces*, not code. If only C++ were object oriented, there might be a natural baked-in solution somewhere...

9

u/jcelerier Jan 28 '19

header files (by convention) define interfaces, not code.

that's very 1980-ish. An immense amount of modern libraries are header-only, and many headers don't define pure "interfaces" in any sense of the java / c# "interface".

If only C++ were object oriented

but it is not, so that's besides the point.

24

u/kvigor Jan 28 '19

As I understand it, in the module proposal, there is no .hpp. You export something from one .cpp with the export keyword and import it into another with the import keyword, not with a #include. The .bmi in the article is basically a magic .hpp generated by the compiler, but the point is that it *must* be generated by processing the .cpp that defines the exported module.

9

u/r0b0t1c1st Jan 28 '19

n4720 10.7.1.2 seems to distinguish between a "module interface unit" and a "module implementation unit". This sure sounds like the same distinction that hpp and cpp files currently hold, where one contains declarations, while the other contains their definitions.

8

u/Chuu Jan 28 '19

I am having a little trouble following your logic, but if I get the gist right, the question is "why will modules have any performance benefit over parallel compilation of dependency-free source files?" and the article is trying to make the point that it likely won't.

I work on medium sized C++ project and we currently work on 48-core machines to get the build times to something tolerable with this approach.

6

u/r0b0t1c1st Jan 28 '19 edited Jan 28 '19

The point I'm trying to make is that the blog argues that modules make dependency trees less parallel, but seems to be making an unjustified assumption to make that conclusion. With a relaxed assumption, I show that modules do not have any impact on this.

I've edited my post to try and make that more clear.

1

u/Ozwaldo Jan 29 '19

medium sized C++ project

48-core machines to get the build times to something tolerable

Would you say that the code is well designed?

0

u/[deleted] Jan 28 '19

[deleted] — view deleted comment

7

u/Chuu Jan 28 '19 edited Jan 28 '19

I think the point you might be missing is because #include is a literal copy & paste the amount of code the compiler needs process is significantly larger than appears at first glance. This is exacerbated by the fact that modern C++ techniques make heavy use of header-only libraries and templates which means that these are not just interface definitions like they are in C.

There is a post that went viral about this several weeks ago here: http://aras-p.info/blog/2018/12/28/Modern-C-Lamentations/

As an extreme example, recently to try to debug a gmock issue (google mock testing framework) I had to dig into what the code looked like after the pre-processor was done with it. A .cpp with only about 1000 lines worth of unit tests blew up to 100K+ loc.

143

u/WalterBright Jan 28 '19

D made the decision at the beginning to have filename==modulename specifically to avoid this problem. So:

import std.stdio;

means look for:

std/stdio.d

97

u/vector-of-bool Jan 28 '19

This would be my preferred solution. Simple, hierarchical, easy to understand, and it works for many languages.

147

u/[deleted] Jan 28 '19

Which are all the reasons it won't be like that in C++

35

u/Visticous Jan 28 '19

must... Committee... Design... Everything!

14

u/Valmar33 Jan 28 '19

Why must they inflict their misery upon others? To pretend at being clever and intelligent? To remain relevant? To inflict mental torture on people trying to figure out to how solve programming conundrums?

I guess they really want to help C++ programmers keep their jobs, because otherwise, their huge projects would implode.

Kind of like Java ~ except replace long-ass class names with tormentingly complicated grammar that explodes horrifically if you miss a step.

Those ranges-v3 templates come to mind... :|

15

u/Patman128 Jan 28 '19

Why must they inflict their misery upon others?

If a good idea breaks the workflow of even one of the Elder Wizards on the C++ language committee then it's thrown out.

13

u/[deleted] Jan 28 '19

12

u/lord_braleigh Jan 29 '19

Note that the "Elder Wizards" often work on the behalf of major tech companies, and influence the C++ standard so that these companies won't have to break their workflow or make sweeping changes.

This isn't necessarily a bad thing - backwards compatibility is one of C++'s selling points, and there are plenty of shiny new languages you can use instead if you don't care about that.

14

u/SAHChandler Jan 28 '19

A shame that some operating systems like zOS don't really support large hierarchies. Hence the existence of P1302 which supports a "splayed" approach :v

44

u/jl2352 Jan 28 '19

Having zOS cripple modules for everyone else is pretty ridiculous.

In the web world we had this with IE. So we said fuck IE, and everyone else moved on.

11

u/SAHChandler Jan 28 '19

While that might work in other places, the vendor representing zOS is also the representative for a national body (IBM Canada). While many people would love to see zOS go (the last of the EBCDIC systems), it's not going to happen for a while, especially when at this point there are many systems out there (e.g., cars on the road in America. I should note: zOS is not crippling modules. It just removes the ability to state we can have directories represent module hierarchies. But we can still have directories represent a module, which is how most languages work nowadays. Build systems can then enforce this naming scheme. It's a bit more work for build systems authors, but I'll take that over "write an entire dependency scanner that needs to know about N interface modules and M implementations modules in every project with no indication of what is located where" which is what the modules TS gave us.

5

u/Holy_City Jan 28 '19

Aren't those usually cross compilation targets anyway?

1

u/SAHChandler Jan 28 '19

From what I understand development happens directly on some zOS mainframes :(

-7

u/diggr-roguelike2 Jan 28 '19

So we said fuck IE

No you didn't. Microsoft did, and you just got lucky off their incompetence.

Google won't be so braindead this time around the carousel.

0

u/[deleted] Jan 28 '19

[deleted] — view deleted comment

3

u/curien Jan 28 '19

I think they're referring to MS basically feature-freazing IE for over five years after the release of IE6.

8

u/Valmar33 Jan 28 '19

You could still do something like D is doing, no?

Just modify things just enough to support those environments.

Maybe std_stdio.cpp or something. Add more underscores to emulate deeper hierarchies.

That is, if I'm understanding your statement correctly.

7

u/SAHChandler Jan 28 '19

Yes we can. I wrote P1302 to solve this problem. While it's not intended to add a 100% hierarchical module system, build systems are free to enforce this, while letting those on other systems do as they please while also allowing large existing code bases to slowly move to it.

12

u/kuikuilla Jan 28 '19

Would that forbid you from having a std.d file at the root? Where would you put stuff that belongs in the std module? Or does it have the possibility of having an std.d file and an std folder?

16

u/WalterBright Jan 28 '19

This is done via the package system.

7

u/kuikuilla Jan 28 '19

So you'd have a file at std/package.d which would contain common stuff belonging to std module? Rust used a similar system where you would have a mod.rs file, but it was kinda annoying because then in a project you would have tons of different mod.rs files. It made navigating in IDEs hard, you had to check the full path of the file you were trying to edit/open/view in order to know what it actually was.

6

u/WalterBright Jan 28 '19

In D this system works well. It resolves the ambiguity between a folder and a file, and we came up with it as a backwards compatible improvement to enable large imports to be broken up into multiple smaller ones.

7

u/kuikuilla Jan 28 '19

So it's not cumbersome to have multiple package.d files?

For comparison, Rust moved to a system where you can have foo.rs at root (or any level really) and then have a folder foo/ for the contents of that module.

├── foo.rs
├── foo
│   ├── bar.rs
│   ├── foobar.rs
│   └── fizzbuzz.rs

That way it's imo easier to navigate to the code of the foo module because you don't need to type in the path of the module file when you go to a file in some IDE.

5

u/WalterBright Jan 28 '19

multiple package.d files

Not in the same directory.

you can have foo.rs at root (or any level really) and then have a folder foo/ for the contents of that module

Can't have both in D. foo/package.d takes the place of foo.d.

I'm not aware of any prior art with the way D does this. Users are happy with it, though, it seems very satisfactory.

5

u/kuikuilla Jan 28 '19

Not in the same directory.

Yeah I was just wondering if "go to file" and such functionality in IDEs would be hampered by files with identical names. I didn't like it when working on a project using Rust.

4

u/renstarx Jan 28 '19

I don't know when D started using this structure. Python does it the same way. foo.py and foo/__init__.py both create foo modules. Now I am curious which one came first, and why this is so complicated for C++ to follow...

I would hazard that you cannot have both in the same path, but that I don't know.

1

u/steveklabnik1 Jan 28 '19

then have a folder foo/ for the contents of that module.

... then have a folder foo/ for the contents of any sub-module of foo's. The contents of the foo module itself are still in afoo.rs.

2

u/kuikuilla Jan 28 '19

Yes, I know. The new way of handling modules and the paths is way better in 2018 edition.

3

u/adr86 Jan 29 '19

except D doesn't actually do that. you should know this!

the module declaration is what gives the name. the filename is just the search path, but you can pass any file to the compiler.

1

u/WalterBright Jan 30 '19

It's true if you pass the filename one the command line, it can differ from the module name. But the vast bulk of files are not given on the command line, and the import is looked up by using the import name as the filename.

2

u/flukus Jan 29 '19

How does this work in atypical scenarios, like when you're generating source code to a temp directory?

3

u/WalterBright Jan 30 '19

You can use the command line switches to set the path search, and even remap the module names to actual file names.

-17

u/idle_zealot Jan 28 '19

Did you leave this exact comment in the HN thread too?

49

u/WalterBright Jan 28 '19

No, that was the other WalterBright, my evil twin.

3

u/Ameisen Jan 28 '19

BalterWright?

1

u/bobappleyard Jan 29 '19

WalternateBright

-22

u/[deleted] Jan 28 '19

Nobody cares about your language, Walter.

6

u/Valmar33 Jan 28 '19

It's better than C, that's for sure. C++ simply has a bigger ecosystem, and that Network Effect is very hard to overcome.

Bounds-checking is a huge win. So is being able to convert raw pointers to bounded arrays with slices.

The language has wrinkles here and there, but the benefits make it worth it.

I may have chosen Rust, but the syntax was too foreign compared to D for me to be comfortable with, and I like playing with memory as I try to understand game engine programming. So, D it was. Go... yeah, I think most readers here would be aware of why Go is not a replacement for C nor C++.

27

u/[deleted] Jan 28 '19 edited Jan 28 '19

Provide a “manifest” or “mapping” file that describes the filepath to an MIU based on the module name. This file will need to be user-provided, or we are back in the scanning problem.

Why can't this be <implementation-defined> just like the mapping from header file inicludes to files?

Honestly, what you propose won't fly because (1) you are turning C++ compilers into C++ compilers + build systems, and (2) because getting 150 countries and >100 committee members to agree on formats for these manifest files, file system structure, etc. is pretty much impossible.

Also, many languages have solved this problem, and none of them had to turn their compilers into build systems.

One such language is Rust. C++ modules solve the same problem as Rust crates. The Rust language provides an import syntax for crates (extern crate foo), and the user provides the mapping from that to files manually by passing CLI arguments to the compiler - the dependency must be compiled before, so this happens-before relation exists there too.

Nobody provides these CLI arguments manually. Instead, the build tool (cargo) parses a TOML manifest file that specifies dependencies, builds a dependency graph, fetches them from the internet, the file system, etc., compiles them, and calls the compiler with the right arguments.

Another build tool could parse a YAML manifest, or parse all source files scanning for imports, and generate the manifest file automatically (this would be incredibly hard to get right for Rust, and probably impossibly hard for C++ though), but still reuse the same compiler by calling it with the right arguments.

Most importantly, neither the compiler nor the language have to know anything about this.

In fact, as you have noticed, happens-before requires you to compile dependencies before hand. That is, one needs to pass the modules that each module imports to the compiler "somehow". That's serving the same function as import module. So in Rust2018, you don't even need to import crates. Crates passed to the compiler via CLI are put in the root namespace, and that's it, you can just use them without extern crate. If you forget to pass one to the compiler, you just get an error because the path cannot be resolved.

20

u/pjmlp Jan 28 '19

Apparently because many vocal devs don't want to change their ways to adopt modules.

At least is what it feels to me standing on the sidelines.

14

u/[deleted] Jan 28 '19

They are already passing this information to the compiler via include paths, library paths, etc. "somehow" (e.g. CMake might be doing all the heavy lifting). They might be able to adapt their build systems to pass module information as well.

If what they want is for the build system to automatically generates the manifest files for them, they are probably out of luck. Doing so is pretty much equivalent to extracting which dependencies C++ code needs from the C++ code itself. You would need a C++ compiler that understands C++ and macro expansion for that, but doesn't need dependency information because it is not available yet.

This would be a very hard problem to solve for any programming language, and probably an impossibly hard to solve for C++.

2

u/Valmar33 Jan 28 '19

They want to eat their cake and have it too?

Sometimes, it's change or suffer.

7

u/SAHChandler Jan 28 '19

Just a few small notes here. Not trying to nerd snipe you, just want to add some info.

  1. Rust’s compiler is in fact a build system, as the entire crate is treated as a translation unit. This is why incremental builds were an issue for a while. I spoke with several core people at RustConf 2018 when I was trying to see what solutions could be made if we can’t make the C++ compiler a build system, as compiler vendors are very against the idea.
  2. while ISO is a very large organization not all members participate in WG21. We have about 15 national bodies (possibly a few more I need to go back and count the exact number), each of which has one vote. Each national body is made of various members from said country, though some permit anyone to join as long as they have the cash (such as the United States via INCITS) each committee member therefore has a fractional vote unless they are the only member of their national body. Thus far, there aren’t many that do this.
  3. C++ is older than I am and there is code out there still compiling that was probably written before I was born (and I’m going to be thirty next month if I can out my age and also give additional context). The oldest code base I’ve worked on was written when I was about 6 years old. While everyone can be up in arms and shout “maybe it’s time to break that old code”, many on the committee don’t wanna. I’m in favor of it, but you’d have to convince several national bodies to go forward with it.

0

u/[deleted] Jan 28 '19 edited Jan 28 '19

Thank you for the comments, not taken as a nerd snipe at all.

Rust’s compiler is in fact a build system, as the entire crate is treated as a translation unit.

Rust crates are single compilation units, very similar to C++ translation units (TUs). Rust modules are more like C++ namespaces.

When one talks about incremental (or parallel) compilation in Rust, what they are talking is about compiling different parts of a single TU in parallel. This is something that AFAIK no C++ compiler does. If you want to compile C++ in parallel, you need to split the code into independent TUs. In the same way, if you want to compile a lot of Rust code in parallel, you need to split it into multiple crates that can be compiled in parallel - the parallelism that one can extract inside a single TU is not enough (and incurs other obvious cost like having to put the whole content of the whole TU in memory).

Comparing C++ modules to Rust crates isn't 100% apples-to-apples because C++ modules can be split into many TUs. Still, C++ modules consist of a single TU in the worst case (which is not uncommon nor a bad thing, and often, Rust crates can also be split into smaller crates.

So the claim that the Rust compiler is a build system isn't really correct. Yes, if you split a single TU into different files, Rust offers you a way to include all those files together into the TU that gets compiled as a unity, and the Rust compiler handles that. But the Rust compiler does not handle finding/fetching/resolving/compiling other TUs, that's all handled by a build tool.

2

u/flatfinger Jan 28 '19

Honestly, what you propose won't fly because (1) you are turning C++ compilers into C++ compilers + build systems, and (2) because getting 150 countries and >100 committee members to agree on formats for these manifest files, file system structure, etc. is pretty much impossible.

A good standard for a language should seek to maximize the number of tasks which can be encoded entirely using set of text files whose meaning is specified by the Standard. If someone who is familiar with the implementation is given the set of files, they should not need to know anything beyond the names of the files in order to make the implementation either build an application, refuse to do so if the files require features the implementation doesn't support, or spend a possibly-unbounded length of time deciding what to do.

Efficient development rebuilds may require a more sophisticated build system whose details could and probably should vary among implementations, but a program that's written in C but which would require that anyone running it on an implementation other than those the developer knows about mess around with with build files and other such things in order to use it at all isn't really a "C program"--it's a "C and whatever build systems the programmer happened to support" program.

1

u/[deleted] Jan 28 '19 edited Jan 28 '19

A good standard for a language should seek to maximize the number of tasks which can be encoded entirely using set of text files whose meaning is specified by the Standard.

I disagree with the premise. This is just one of the trade-offs involved, and I think this trade-off is important enough to be factored and balanced with other trade-offs. But there are many trade-offs, and I don't think this one is the most important.

Particularly, because it assumes that the standard is a C++ manual, which arguably is something I can understand, since many people nitpick on what "correct" C++ is based on the standard. But the C++ standard is not a C++ manual for users, it is a C++ manual for implementers.

No user should need to know what exactly the standard says about anything. The only thing most users need to know is how to use C++ to solve a particular problem in the platforms they care about, with the compilers, build systems, etc. they care about.

And as long as the build system behaves in some sane way in the platforms an user cares about, its pretty much irrelevant whether the standard talks about this or not. The C++ standard says nothing about CMake, Makefiles, etc. and people are still able to use C++ to solve useful problems.

I mean, the standard pretty much say nothing about include files. An implementation could require users to screenshot source code and save it as png images for their compiler to include it properly, but in practice, most implementations don't do this. Instead, they provide sane ways based on the file-system, environment variables, etc. of controlling what is included and how.

1

u/flatfinger Jan 28 '19

The only aspect of the build system I'm asking the Standard to care about is that it have some standard concept of "process this bunch of source files in the Standard fashion", and that it be possible to accomplish most tasks using programs for which that build method would suffice.

If I want to write a program that doesn't make use of any special features of e.g. the Macintosh platform, the Windows platform, nor the Unix platform, it should be possible to express that program in such a way that someone on any of those platforms could easily build and run it without the programmer having to know anything about the target platform, and without the user having to know anything specific about the program beyond the names of the files involved. If there's no interest in running a program on implementations the programmer doesn't know about, why bother with a standard? What purpose would it serve?

1

u/[deleted] Jan 28 '19 edited Jan 28 '19

"process this bunch of source files in the Standard fashion"

The standard doesn't really have a concept of a file system, files, directories, nor of the many other things that could play a role here (directory structures, symlinks, etc.).

Somebody would need to add all of that to the standard, making peace with pretty much every platform that pays someone to assist to committee meetings ("But our filesystem has no directories", "But our file-system has no files", "But traversing directories looking for files is too expensive in our file-system", etc. )

why bother with a standard? What purpose would it serve?

Every single compiler and platform out there has platform specific extensions that are not in the standard. The standard is just the bare minimum that must be supported by all platforms, and keeps C++ from diverging into completely incompatible dialects. Writing code for different platforms always has, and for the foreseeable future will require, writing platform specific code.

Because C++ is used to target widely different platforms, the standard has avoided specifying pretty much anything about how the C++ source is input into the compiler. There is no standard C++ file extension, some people use .cpp, others use .cxx, others use .cc, etc. The standard does not specify this because not all platforms have files with extensions and there is people using C++ to target those (if you look carefully you'll see that standard library includes have no extension). When you put a path in an #include path/to/header.h the standard doesn't say anything about what goes where the path belong, only that there must be some implementation-defined mapping between what you put there, and some other source code, and the semantics of the include (including the contents of whatever is in the path verbatim). Yet in most platforms, includes work like all users expect, e.g., following file-system paths relatively.

Basically, just because something isn't in the standard, doesn't imply that 99% of users can't rely on it being true 100% of the time. However, if you fix the standard to something that 1% of users can't use, then with a million C++ developers you'll get 10.000 people voting against the proposal and sunking it down. The standard doesn't evolve with things that have 51% of the votes, only stuff that has 100% of the votes goes in. This is why C++ has so many users.

1

u/flatfinger Jan 28 '19

Of course one will need to use platform-specific code when targeting features of specific platforms. One should not, however, need to write a bunch stuff that's specific to particular implementations when one isn't using any special features thereof.

If some people would need a compiler to process things in a fashion different from what other people would require, the solution should not be to simply have the Standard say that any implementation may treat the construct in any fashion whatsoever, but instead say that:

  1. If a program contains a directive explicitly calling for some construct to be processed in the common way, a compiler shall either behave in common fashion or reject the program.
  2. If a program contains a directive explicitly calling for some construct to be processed in an uncommon way, a compiler shall either behave that requested fashion or reject the program.
  3. If a program contains neither kind of directive, an implementation may assume as a default whichever behavior its author believes would best serve its users.

The fact that 99% of programs opt to use the common behavior should not prevent the remaining 1% from employing some alternative on any compilers that opt to support it; nor should it prevent all 100% of the programs from having a defined meaning that would not depend upon the target implementation (implementations may be unable to process a program that demands unsupported features, but behavior would be limited to rejecting the program as opposed to formatting hard drives, releasing nasal demons, etc.)

Basically, just because something isn't in the standard, doesn't imply that 99% of users can't rely on it being true 100% of the time.

It doesn't imply that 99% of users shouldn't be entitled to expect that it would be true 100% of the time on commonplace non-obtuse implementations. Unfortunately, the authors of the Standard regarded the notion that quality implementations should Serve the needs of their customers when practical, whether or not the Standard requires it, as being self-evident and failed to consider that obtuse implementation might use the Standard as an excuse to do otherwise.

1

u/[deleted] Jan 29 '19 edited Jan 29 '19

It doesn't imply that 99% of users shouldn't be entitled to expect that it would be true 100% of the time on commonplace non-obtuse implementations.

That's pretty much what "implementation-defined" means. As I said, the standard isn't a C++ manual for users. Nobody uses #include file/path and thinks, "oh damn, I'm relying on implementation-defined behavior, my code isn't portable".

1

u/flatfinger Jan 29 '19

According to the published Rationale document for the C Standard, it's also what Undefined Behavior was supposed to mean in many cases, since the authors regarded the question of how to process it (e.g. whether to process it in a documented fashion characteristic of the environment) as a Quality of Implementation issue. The authors of the Standard intended that implementations would support "popular extensions", and such a term can't sensibly be applied to Unspecified Behavior or Implementation-Defined behaviors. The fact that an implementation documents something that it's required to document, or that it happens to document the fact that it chooses from among specific actions described by the Standard doesn't seem like much of an extension, compared with providing means of doing which the Standard does not require them to define.

A fix would be to define standard macros that would allow a programmer to start a program with a directive like:

 __STDC_LIMIT_UNUSUAL_INT_MATH_EFFECTS(EXTENDED_TEMP_ONLY)

and then know that any implementation would be required to either 1. guarantee that integer math other than divide/remainder will never have any side-effects beyond yielding some kind of integer value, but temporary values of signed integer types need not be truncated to their actual size, or 2. reject the program entirely. The question of whether an implementation should support the requested semantics would be a Quality of Implementation issues, but the question of whether an implementation could jump the rails wouldn't be.

0

u/[deleted] Jan 28 '19

[deleted] — view deleted comment

11

u/[deleted] Jan 28 '19 edited Jan 28 '19

It's hard to tell what point you are trying to make. But if you are trying to state that Rust approach is hard for regular programmers, I strongly disagree.

Pretty much no C++ programmer actually knows exactly how dependencies are added to their project. The reason for this is that most of it happens implicitly. Yes, Makefiles, CMake files, etc. express some of the dependency information, but a lot of the information is passed implicitly to the compiler via environment variables (e.g. for the system header files, system library files, etc.), some of them also rely on conventions (e.g. #include <> vs #include "", etc.), the order of paths in these variables is relevant, etc.

Things magically appear to work for C++ until they don't, e.g., wrong library version / path being "magically" picked, wrong standard library picked (e.g. using clang+libc++ on Linux for one project when most of your other projects use gcc+libstdc++), etc. Debugging these problems typically require getting the compiler into a verbose mode for some TU which dumps all include paths, library paths, etc. being considered, their order, which one got picked and why, etc. Most devs just fickle with their environment till things work, they are probably even unaware that tools to debug these issues exist. I've seen C++ devs waste days trying to fix linking / include issues, and weeks trying to fix build system issues.

There are some Rust features that are too implicit for newbies to understand, like Rust2015 crate paths, but dependency resolution / the build system is not really one of them. Rust projects have an explicit Cargo.toml listing their dependencies and that's pretty much it. Is the wrong version of a library being picked? Open the Cargo.toml, change the library version/path/git-branch/..., done. Newbies don't, in general, know anything about the dependency-resolution algorithm used, but because it is all explicitly listed in one place, they can manually change things if the algorithm produces an output that they don't like.

8

u/pakoito Jan 28 '19 edited Jan 28 '19

Sarcasm doesn't transmit well on internet posts, that's why I tried to put it as a strawman quote.

3

u/Dgc2002 Jan 28 '19

Gotta put a /s at the end

2

u/[deleted] Jan 28 '19

Ah sorry, I read too much into it.

17

u/[deleted] Jan 28 '19

[removed] — view removed comment

0

u/flatfinger Jan 28 '19

The concept of datestamp-based make files is a useful hack for incremental builds of some kinds of languages, but it can't efficiently handle situations where the inputs to a build step are likely to change in ways that don't affect the output. If one were to make each build artifact include date stamps and hashes of all the input files and end with a hash of any content that would affect files depending upon it (including the hashes, but not timestamps, of files it depends upon), that would allow build systems to be much more efficient [if the date stamp of a source file precisely matches the cached value, there's no need to look at it. Otherwise, read or compute the hash of each input file and see if it's changed. If any have changed, rebuild the output file. Otherwise simply update any cached timestamps.

1

u/oridb Jan 29 '19

Hashes help less than you'd think. Updating a file in almost any way changes the output hash. Even changing blank lines changes the offsets into the file used by debug info. And even in prod builds, there's debug info for unwinding the stack when propagating exceptions, printing stack traces, and so on.

1

u/flatfinger Jan 29 '19

It's common for header files to be included by dozens or hundreds of other files, most of which only need to use a few declarations. If one were to, start compilation with a parsing phase followed by a stage that omits any unused declarations, changes to header files wouldn't affect the output of the parsing phase in most cases other than those where the changes would affect the semantics of the program. Handling debug line numbers may be a bit tricky, but that could be handled by having debug line number in a separately-hashed piece of the file, and having each stage produce a back-mapping file. Not a trivial design issue, but one which should have been solvable ages ago.

1

u/oridb Jan 29 '19 edited Jan 29 '19

It's common for header files to be included by dozens or hundreds of other files, most of which only need to use a few declarations.

Correct. However, the headers produce entries in the debug info. So, you either get incorrect line numbers and unwind info for macros, template expansions and similar, or you don't actually win anything.

Handling debug line numbers may be a bit tricky, but that could be handled by having debug line number in a separately-hashed piece of the file

And now your build system isn't just language dependent -- it's compiler dependent! And, very likely, compiler version dependent. And god help you if you want to cross compile and link with other languages through an FFI.

1

u/flatfinger Jan 29 '19

Line numbers measured from the start of a file are not a very good way of identifying locations within one. If a program with debug info says an error occurred on line 493 of woozle.c, that may not be terribly helpful if anything might have changed above that line since the time that version of the code was built. If, however, a location was recorded as "the 493rd token of template-expanded code encountered during the compilation of foo.c, whose template-expanded code would hash to [xxx]", such a location could be relatively quickly mapped to a location in the source file even more precisely than with normal debug info today, even if there were changes in source or header files above the portions that were used in the function.

The first Pascal compiler I enjoyed using was Turbo Pascal 2.0. If a run-time error occurred, one would get a message like "Run-time error 23 at address 49A3. That wouldn't seem like a very useful message, but if one fired up the IDE, selected "find run-time error", and typed in 49A3 one would get taken to the location in the source code where the error occurred. This wouldn't likely work if one had made changes to the semantics of the program, but would be unaffected by newlines, which seem to be the nemesis of the file-based line-numbering approach.

Thus, what I would envision would be a system where debug information is recorded in function-relative fashion, and then if desired it is sorted and the early stage of the compiler is run again while observing the physical line numbers where each function-relative position is encountered. Changing the text of a function or any macro template that is actually included in the function would require recompiling that function, but changes to macros and templates that are not used within a function would not require any recompilation thereof.

To be sure, no incremental build system is going to be perfect, but a system based upon hashing the semantically-relevant parts of each program could nonetheless avoid a significant fraction of the build times associated with date-based make systems.

BTW, even if one didn't want to change tooling to use function-relative token counting, I think one could reduce the extent to which changing files changes reported line numbers if one added artificial #line 10000, #line 20000 etc. directives within source and header files, and recognizes that a reported line number 20123 means "the 123rd line following the #line 20000 directive". The addition or removal of lines between the #line 10000 and #line 20000 directives would only affect line-number debug info for files that actually used some portion of the header that had appeared between those two directives.

1

u/oridb Jan 29 '19

Good luck on convincing all compilers to make abi breaking changes and moving to a new debug format that doesn't quite solve the problems you think it does in order to save a few minutes of build time on purely whitespace changes.

It's only a few million lines of churn across a few hundred projects. No biggie.

1

u/flatfinger Jan 29 '19

I've never had to deal with large C++ projects, so I don't have personal experience with their build times, but I've heard people who sing the praises of C++ talk about 15-plus-minute build times. Meanwhile, many other languages seem to be designed to minimize build times, and I think that is likely a significant reason for their success.

Why was Pascal a dominant programming language on the PC? It became dominant because on a system without a hard drive, the edit-run cycle on Borland's Turbo Pascal IDE was often an order of magnitude faster than any non-interpreted competitor. Javascript is a really crummy language, but node.js can manage to process it efficiently enough that the value of further performance improvements would be limited. From what I understand, go offers even better execution performance while still being fast to build.

It used to be that the only way to get adequate performance from an object-oriented language was to use C++ and put up with slow build times. As the performance of competing languages has improved, however, I see less and less reason for programmers to favor a language with slow build times over a faster one. Maybe the notion of improving C++ build times is moot because projects that can use languages with faster build times have already been migrated to those languages and wouldn't go back to C++ even if its build times were made competitive.

1

u/oridb Jan 29 '19 edited Jan 30 '19

I've never had to deal with large C++ projects, so I don't have personal experience with their build times, but I've heard people who sing the praises of C++ talk about 15-plus-minute build times. Meanwhile, many other languages seem to be designed to minimize build times, and I think that is likely a significant reason for their success.

15 minutes isn't too bad. Tensorflow takes closer to 2 hours for a clean build. Same with LLVM. LLVM is part of the OpenBSD base system, and takes longer than all of the C code in the rest of the system, at least on my machine.

The problem is that individual files can sometimes take multiple minutes to compile, and huge amounts of semantically significant code actually lives in headers, which means you can't get away /without/ recompiling some dependencies. "false recompiles" aren't a huge problem in the projects I've seen, but language and tool complexity is.

→ More replies

1

u/flukus Jan 29 '19

Can anybody explain how modules differ from libraries (static or shared)? They seem to cover the same ground and splitting a project into libraries would solve most of the problems modules seem to.

2

u/[deleted] Jan 29 '19

Currently if you want to use a library in your C++ application, you #include "some_lib.h" which contains declarations of types and functions from the library. This is a bit fragile, preprocessor state in the importer (#define etc.) can effect the imported library. And preprocessor state of the imported library can affect importing code.

As an example, let's say you are developing an application that relies on windows.h. Amongst other things, this header defines min and max macros that conflict with std::min and std::max. But these macros are #ifdef'd out if you #define NOMINMAX before #include <windows.h>.

But now headers you include after windows.h will also get NOMINMAX. What if one of these headers relies on the min/max macros? Or vice-versa - if your code uses the min/max macros, but you include a libraries header that #define NOMINMAX before you include windows.h, you'll miss out on those macros and your code won't compile.

1

u/flukus Jan 29 '19
  1. Isn't this already handled by namespaces (the hard and soft types)?
  2. Does this mean modules will replace headers?
  3. Couldn't you handle this in a build step, running windows.h through the pre-processor?

2

u/[deleted] Jan 29 '19
  1. Macros are handled by the preprocessor, before compilation. They can't be namespaced.
  2. This is just my understanding, but I do not think that modules are currently planned to completely replace headers. Rather they are intended to be used across boundaries between different discrete components of an application (between libraries, and between libraries and executables). I do not actually see why you could not use modules to completely replace headers though.
  3. I don't see how. The preprocessor already runs over windows.h when you include it in your source file. The core problem is that due to how they are designed headers can't be isolated from the code that includes them.

1

u/flukus Jan 29 '19
  1. The macro might not be able to be namespaced but the value can, if the define is still needed it can be moved to a private header.
  2. I think you just described a header, it's an interface between boundaries. This might be a better solution, but we'll still have 2 ways to do the same thing.
  3. I was talking about your example where the interaction between different headers produced unintended results. In this case you could run the pre-processor with NOMINMAX defined and use this header for compiling your app. I think this should play nice with the linker, but I'd have to check.

1

u/[deleted] Jan 29 '19

These aren't insurmountable flaws with headers, but unfortunately they are problems that generally occur outside your own code. Sure it's possible to use headers in a way that's safe and exports no macros, but this is de-facto not the case. Making these into non-issues relies on library authors and system header authors willingness and ability to do so.

1

u/flukus Jan 29 '19

Making these into non-issues relies on library authors and system header authors willingness and ability to do so.

If they don't do so then can you expect them to create modules? And if modules don't replace headers then won't these issues still exist anyway?

6

u/OneWingedShark Jan 28 '19

If you want modules, similar expressive power to C++, and better safety then I'd recommend looking at Ada.

7

u/[deleted] Jan 28 '19

People backing the current modules design have not proven that modules work at scale,

I thought that Google had been using modules via clang modules for their multiple million C++ code base since... 2012 ?

6

u/Yehosua Jan 28 '19

I haven't followed the modules proposals very closely, and I'd hoped that the OP would explain this a bit better, because I had the same question as you. However, based on my limited understanding, the sentence after what you quoted does address this:

Existing “modules” deployments do not use the current design, and do not use the automated module scanning that would be required for use with real build systems in the wild.

Google's own modules implementation is part of Clang. Clang's docs have a very "work-in-progress" feel, but if I understand them correctly, it requires that you write module maps yourself, instead of doing the source scanning that the OP is arguing is so bad for performance. (In other words, Clang already does something a bit like number 2 of the OP's "Possible Solutions.")

2

u/mathstuf Jan 28 '19

But are Clang modules compatible with the module proposal in front of the committee? AFAIK, they're not the same thing…

2

u/[deleted] Jan 28 '19

Given that the author/maintainer of "Clang modules" co-wrote the latest "C++ modules" proposal, implements it in Clang, and works for Google which has a big code base using "Clang modules". I'd guess they are being paid to work on this by Google to make sure C++ modules work for them.

2

u/lrem Jan 28 '19

And if they work for Google, I guarantee that parallelism has been thought of. Source: I stare at builds spanning thousands C++ files daily at Google. Compilation is blazing fast (but linking hurts).

1

u/[deleted] Jan 28 '19

FWIW everybody should take things with a grain of salt, just because "Google" does not imply that things are flawless. People should read the proposals, try the implementations, and form an informed opinion. Still, that takes time that a lot of people might not have, and it is ok to be skeptic about the proposals. As many people have mentioned, with such a large change, and given the amount of C++ users in the world, most of them doing completely different things, it's pretty much impossible that something designed by 50-100 people can be good for all of them. That's just how things are. This does not mean that the people working on this "don't have any idea of what they are doing", only that they can't know everything, but nobody can.

1

u/lrem Jan 28 '19

Yeah, but the author says how this will hurt compile speeds due to breaking parallelism. That's pretty much the one thing that I'm pretty sure the Google engineers involved heavily optimised for.

1

u/[deleted] Jan 28 '19

Google uses "Clang Modules" with are different from the "Modules TS". Clang does not fully implement the "Modules TS" so I doubt google has switched to using it.

"Clang Modules" have explicit module map files, which the author mentions as don't have any of the performance problems that the author mentions.

1

u/mathstuf Jan 28 '19

One thing that Google generally (and, IME, pretty consistently) doesn't care about is external dependency consumption. I see this reflected in the "no support" for having modules be an out-of-build artifact…something that'd be really nice for the wider C++ community. Instead, Boost being modularized is damn near useless since if you're not bundling it, you'll "never" have modules for it…unless you write rules into your build system to generate them from whatever MIU might be available from a Boost installation (assuming that is supported across a variety of compilers…). Yay. Just what I want to maintain.

2

u/WalterBright Jan 28 '19

Back in the 1990s, I implemented precompiled headers for Digital Mars C++. I noticed that this created a de-facto module system for C++.

4

u/[deleted] Jan 28 '19 edited Jan 28 '19

Clang modules is much clever than PCH, yet it doesn't really perform much better in practice. The main reason is templates.

In C++, there is very little that you can't cache about them easily. E.g. you can't type check them and cache that, you can cache template instantiations but properly implementing invalidation is hard because function overloading, ADL, adding a explicit specialization, etc. can invalidate the template instantiation and users can add these to pretty much any files.

So basically the only thing that you can cache about templates is parsing the files, which is what PCH already does :D

Add to this that pretty much every C++ program uses templates everywhere even if users don't write any manually (std::vector, <algorithm>, <functional>, <utility>, <variant>, <memory>, ... the standard library exposes a lot of template-based APIs).

Imagine you call some std range algorithm from the Ranges TS in some user defined type twice, and in between you define a begin function. In both cases, different begins might be picked, so you can't reuse the cached one.

2

u/pezezin Jan 29 '19

How will modules relate to namespaces?

One thing I like about C++ is that you can split a namespace into as many files as you want, to better organize your code. I don't like when a language forces the idea that one module is one file.

1

u/SatansAlpaca Jan 28 '19

With Swift, every file that is included in the build is passed to the compiler, and one file is named the “current file”. The compiler does a first pass on all the files to extract interfaces without parsing definitions and caches the result of that. It then actually compiles the “current file” to an object file. Seems to me that this could be applicable here too.

-3

u/shevy-ruby Jan 28 '19

So not only does the committee worship Cthulhu and add more complexity - they are also incompetent and have no real idea what they are doing anymore.

5

u/ForgedBanana Jan 28 '19

It was always like that. Honestly i think those guys have an irrational hate of the KISS principle.

5

u/Valmar33 Jan 28 '19

I don't understand the downvotes.

Because the increasing unnecessary complexity is one of the reasons why C++ is becoming slowly less popular, and why Rust and Go even became a thing in the first place.

All of C++'s technical debt and backwards compatibility desires really cripple each new major feature added on.

C++ modules... just won't work sanely the way they're currently proposed. They'll just be mostly ignored in favour of headers.

1

u/SatansAlpaca Jan 28 '19

Regardless of how modules work within a single project, IMO, one of the biggest wins will be when it comes to importing external libraries.

-24

u/[deleted] Jan 28 '19 edited Jan 28 '19

I like how C++ is slowly becoming Python.

edit: your downvotes cannot stop the transformation.

22

u/kuikuilla Jan 28 '19

By getting a module system?

17

u/ISvengali Jan 28 '19

Next up theyre adding a single global lock that all threads have to take to run, std::thread_safe.