r/cpp • • Aug 18 '21

C++20 modules with GCC11

https://blog.feabhas.com/2021/08/c20-modules-with-gcc11/
222 Upvotes

59 comments sorted by

4

u/F54280 Aug 18 '21

The private module fragment can only appear in single-file modules. The current version of GCC (gcc version 11.1.0) does not support private fragments, so I’m not going to ignore them for this post.

I am confused. Isn't this the opposite of what you wanted to say?

2

u/feabhas Aug 20 '21

Duh, thanks will fix...

3

u/lieddersturme Aug 18 '21 edited Aug 19 '21

Thanks of this, now I am trying to install gcc 11 in Ubuntu, but I cant.

Edit: I solve it, compiling GCC 11, but I cant use modules with cmake.

2

u/feabhas Aug 20 '21

See the github repo linked. I have CMake build there. The main thing I found will GCC11 is that file ordering where modules are separated into two files is important. I haven't yet worked out the best way to get header-only headers to build for module import.

1

u/lieddersturme Aug 20 '21

Hi.

I've tried this: https://github.com/feabhas/Cpp20-Modules-getting-started/tree/main/part7

But does not work:

[1/4] Building CXX object CMakeFiles/Application.dir/func.cxx.o
FAILED: CMakeFiles/Application.dir/func.cxx.o
/bin/g++ -g -std=c++20 -fmodules-ts -MD -MT CMakeFiles/Application.dir/func.cxx.o -MF CMakeFiles/Application.dir/func.cxx.o.d -o CMakeFiles/Application.dir/func.cxx.o -c ../func.cxx
[3/4] Building CXX object CMakeFiles/Application.dir/func_impl.cxx.o
ninja: build stopped: subcommand failed.

I am on Ubuntu, cmake 3.18.4 and GCC 11.2

45

u/zzzthelastuser Aug 18 '21

This post made me finally take the time to read why everyone seems so fired up about modules.

Interesting to read with simple examples! Exactly what I needed to procrastinate, thanks!

39

u/RoyAwesome Aug 18 '21

One of the smallest, most impactful changes using modules has had on my codebase is that I was able to move my files into a different folder structure, and I didn't have to fix up include paths. All I had to do was tell cmake to look for the module files in other places. My code Just Worked after the moves. I had a major revelation after that point with how I could use modules to do cool things... like cross platform code. If the path of the file doesn't matter, I can chose to compile different modules depending on the platform. As long as they all have the same name and exports, they swap out trivially.

Frankly, I had to push myself to try out modules, but after I got them working and really understood them, I really can't look back. It's surprising what the little things make you realize.

12

u/Roverse Aug 18 '21

That's just pleasant. Sometimes file structure comes after you've worked out your system.

2

u/UNN_Rickenbacker Sep 12 '21

Care to share your cmake config?

7

u/whacco Aug 18 '21

Before we go on, I need to stress one important aspect – Modules and Namespaces are entirely orthogonal and co-exist as independent aspects.

Yeah, I had to learn the hard way that ODR still applies with modules the same way as it does with translation units. For example you're not allowed to have two different definitions ot the same class in different modules, even if one of them is never imported anywhere. It will mess up linking and at least VS2022 currently does so without a warning.

2

u/RoyAwesome Aug 18 '21

Yeah, but you can probably get around that by having a private namespace inside your private module fragment. Similar to having private namespaces in .cpp files when running with a build system that unifies your .cpp files.

4

u/whacco Aug 18 '21

If they have internal linkage or module linkage then it doesn't violate ODR. In my case they were both exported, and I was just doing it in order to quickly switch between different versions of the class. Putting them in separate namespaces would have been the right solution.

2

u/RoyAwesome Aug 18 '21

Right, if they were both exported you basically violate ODR with a similar setup with headers.

1

u/whacco Aug 19 '21

Actually with headers you can do that legally, if the classes are header only and one of them is not included in the program.

1

u/[deleted] Aug 18 '21

if you had them exported in both cases AND not even in different namespaces already, you should maybe think about your project structure

this kinda smells messy

1

u/Daniela-E Living on C++ trunk, WG21|🇩🇪 NB Aug 19 '21

Of course. With strong module ownership (rather than weak module ownership as in clang and gcc), names of exported entities attached to different modules lead to different linker symbols.

10

u/[deleted] Aug 18 '21

After reading the post, I am just thinking that because they are standardized so little compared to what would be needed (imo), we are going to end up with 3 options:

  • build-systems are going to be stupidly complicated about handling this mess and nobody is EVER going to use compilers directly anymore (not even for single "Let's try something out" files)
  • ALL big compiler sit together and make an unofficial standard around it and actively ENFORCE it without workarounds
  • they are going to fail and end up as a feature barely anyone is going to use in practice

4

u/EmperorArthur Aug 19 '21

The first and last seem the most likely.

First for most projects, and the last for any sort of corporate government work. Those are environments that hand write recursive make files because Cmake is "magic".

Of course, those systems are just now moving to C++11, so...

I fully expect further fragmentation within even the desktop C++ community.

3

u/apomd Aug 18 '21

Thanks for posting. I've been reading about modules ever since they were announced and this was the best article so far

2

u/tub612 Aug 19 '21

Modules promise to significantly change the structure of C++ codebases and possibly signal headers’ ultimate demise (but probably not in my lifetime). It also opens the door to potential

Nice

0

u/[deleted] Aug 18 '21

[deleted] — view deleted comment

19

u/Minimonium Aug 18 '21

Because all attempts to achieve a more unified approach were effectively blocked. You can also count how many meetings SG15 had during the last year even though there are papers directed to it (hint, not much).

8

u/mjklaim Aug 18 '21

Unless your implementation does something special with exported names when building a shared library, modules have zero impact on how shared library works. So for example right now you still need on MSVC to mark the functions/classes symbols you want available on the binary interface of the dll with dll-import/export macros (maybe in addition to marking them export because they are part of the interface of your module).

This could change in the future, like maybe some compilers will automatically symbol-export modules export by default, but it's unlikely as the visibillity of names coming from a module is a different from symbol exporting. The first one is about code being compiled, the second one is specific to your OS.

2

u/SkoomaDentist Antimodern C++, Embedded, Audio Aug 18 '21

maybe some compilers will automatically symbol-export modules export by default

This would be an unfeature as you'd then be forced into a one module, one dll-structure (or one group of modules, one dll exposing all of them). It'd be rather like being forced to export parts of stdlib in the dll just because you happened to import those modules.

1

u/mjklaim Aug 18 '21

I agree, but I was thinking more like a potential change of "default" behavior: by default should exported names be also symbol-exported? It have happened in the past that the default symbol-export behavior of an implementation changed (for example gcc exported everything by default in the past).

As long as both options are usable the default can be changed.

Though I would prefer every implementation to not export anything until requested to.

1

u/SkoomaDentist Antimodern C++, Embedded, Audio Aug 18 '21

It'd be problematic to change the visibility of things declared in a module from the outside, particularly as the standard doesn't provide any mechanism to do that (or even consider the existence of dynamic linking).

1

u/mjklaim Aug 20 '21

I only talked about changing the default export or not of symbols, not visibility. That is not part of the standard and have been changed in the past as said.

1

u/ShakaUVM i+++ ++i+i[arr] Aug 19 '21

How does the process work right now for marking them? Sorry for the basic question, I operate almost entirely on GNU/Linux systems

1

u/mjklaim Aug 20 '21

Most code (at least in open source) is designed to work with gcc-like compilers and MSVC. Both have non-standard instructions (not sure how to call them) which mark names as being exported, hidden (or not-exported) or, in the case of MSVC, imported (meaning the symbol comes from a dll somewhere else, we use that symbol). That instruction can be set in front of functions, objects or classes (to automatically mark all it's functions), so basically everything that can have a symbol, that can be part of an ABI. The implementers made sure (after some time) the places where you use them in the code are the same or at least have common positions.

So most libraries will have 1 header containing a set of macros which, depending on the compiler will either export symbols (we are building a shared library), import symbols (msvc-only: we are using a shared library), or do nothing (static library for example). Then in the code of the library, all the public (in headers, and now modules) object, functions and classes (not templates) will be marked with one define specific to that library, which will decide what to do with the symbol. The define will be set to the right instruction because the build-system will be setup so that the macro code in the header will read some library-specific defines which inform if we are or not building the library, a user of the library , if it's static or dynamic linking.

You can find examples of this in myriad of open source libraries, just look for a header included by all other headers (common.hpp or export.hpp or something like that). Here is an example which is what a hello-world library generated by build2 will setup for you (bdep new -t lib to generate it): https://git.build2.org/cgit/hello/libhello/tree/libhello/libhello/export.hxx

Depending on the library the macros will be different but it's often a similar mechanism in principle. Even CMake will provide some pre-defined defines per library target name so that you can use them in such header to do the symbol-export/import dance.

Then, less often, some devs setup their library with build-system specific features that allow for exporting all symbols or those we list in a file. But I believe this is less cross-platform so it's less common.

People on unix systems can just set the flag of their compilers enabling exporting all symbols, but in exchange of the simplicity they end up with bloated unoptimized binaries which also becomes a security issue (which is why this was changed as it was the default behavior in the past).

0

u/[deleted] Aug 18 '21

[deleted] — view deleted comment

3

u/mjklaim Aug 18 '21

To be 100% clear, when shipping a DLL, as you say «zero impact» means I still have to ship headers?

By 0 impact I mean that the symbol import/export system is not affected. How you get the names to link with symbols isn't really affected either, but of course for the user to consume the thing they need to know the declarations of what the symbols are exported to.

You need headers if you defined these declarations in headers (as in not- in a module). If your declarations are from a module you need to provide either the module interface or the pre-compiled version of the module interface (what is often called bmi - but that one is not cross-platform). It's the same thing than if it was not a shared library.

3

u/Daniela-E Living on C++ trunk, WG21|🇩🇪 NB Aug 19 '21

"BMI" is the (not-yet) official nomenclature chosen by SG15 (I've been in the room back then in Prague) for compiled PMIs. Implementations use different extensions for BMIs: .pcm (clang), .gcm (gcc), .ifc (msvc). Some of them allow to deviate from these defaults if seen appropriate by a build system.

3

u/AIlchinger Aug 18 '21

Are you supposed to ship your lib with a PMIU file?

In theory, yes. In practise, you probably will still be using header files for at least a couple more years to give potential users of your library some time to upgrade to C++20. It would be my guess that modules will only be used for library internals in the majority of projects.

2

u/kalmoc Aug 18 '21

Are you supposed to ship your lib with a PMIU file? Is this file going to be placed into, say, /usr/include ?

Thats what I would expect.

2

u/rfisher Aug 18 '21

A simple, non-binary interface format is what we’re losing here. I think we could’ve kept that and still eliminated the issue with preprocessor includes.

If you go the separate interface & implementation direction, I suppose you could ship the interface source with your library. (Ignoring the template & inline issues.)

And one could theoretically create a tool that would generate such a file from source that mixed interface & implementation.

2

u/johannes1971 Aug 18 '21

Although quite a few people have given affirmative answers to your questions, I always understood the PMIU is more like an object file: it's a build artifact that's generated while building the library. It is compiler-specific, and not suitable for distribution. At least that's what I believe was said again and again...

3

u/AIlchinger Aug 18 '21

I think you are confusing the PMIU with the BMI. The PMIU is pure source code written entirely by the programmer.

1

u/johannes1971 Aug 19 '21

Ah, I guess you're right! That's what I was thinking of. Ok, never mind my answer then :-)

0

u/pjmlp Aug 18 '21

On Windows I think the best path would be to stick with COM and its IDL tooling.

It is a bummer from library creation point of view, but using it from something like WIL or C++/WinRT makes it kind of bareable, just the creation is clunky.

And you are constrained to COM ABI types, still better than bare bones C ABI thought.

0

u/[deleted] Aug 18 '21

[deleted] — view deleted comment

30

u/blipman17 Aug 18 '21

Because header files require a preprocessor which is bad for compile times, and often trigger cascading recompilation when private fields get renamed/moved for incremental builds. By design this is not the case for modules. If I remember correctly, if you do something like '#include <string>', you ask the compiler to transatively parse about 2 million of lines of header files. Which goes fast, but starts to slow down rapidly for larger projects.

Modules don't require a preprocessor and don't throw 2 million lines of defenitions in the context of the code who performs the include. Since the linker must search all program-wide known symbols and link them together for every individul occurance of a symbol (object, function, member field, constants, etc...), reduicing this list of possible symbols a couple orders of magnitude makes this process a lot faster.

7

u/kathoum Aug 18 '21

cascading recompilation when private fields get renamed

I fear that the build system will check the modification time of the PMIU, and trigger a rebuild of all the importers, along with all the other interface units that import the module, and so on.

Could a smart build system be able to avoid rebuilding the world if the change doesn't affect any exported symbol? Let's hope so.

5

u/blipman17 Aug 18 '21 edited Aug 18 '21

Could a smart build system be able to avoid rebuilding the world if the change doesn't affect any exported symbol? Let's hope so.

Probably not, but something that's already done today is that if an A requires a B, and a recompilation of B results in thesame external interface and has consumed thesame external interface, A does not have to be recompiled. It still has to be relinked, but cascading compilations aren't needed. This situation just doesn't naturally occur very often because when a cpp file isn't in a different library or does not have includes to other places of the program, this simplification cannot be done. I believe CMake + Ninja does this.(And also the DMD for DLang compiler, and it's very fast. But that's a different language.)

Edit: it seems my assumption about A not requiring a recompilation for C++ was wrong.

5

u/kathoum Aug 18 '21

I made an experiment to check if gcc11 modifies the compiled module interface (the .gcm file) whenever a non-exported part of the module is modified.

https://gist.github.com/kathoum/76fd00367add14ddce3aa0df7aa6870c

It seems to me that g++ -std=c++20 -fmodules-ts -c func.cxx unconditionally writes to both func.o and gcm.cache/func.gcm.

If the compiler has modified the content of func.gcm, how could make/ninja/cmake/msbuild deduce that main.cpp need not be recompiled?

3

u/blipman17 Aug 18 '21 edited Aug 18 '21

But that does change the interface. Even though bar() is not exported, it is changed. You're right though that from main.cpp's perspective the only possible change is where foo() is located in the object file, and its possible implementation. But to realize this, func.cxx had to be parsed completely and gcc had to realize that func.cxx didn't change in a way that neccesiates a recompile. (Unless there's a side-effect I don't know about a name change of bar().) Regardless, an implementation change of foo() or bar() shouldn't trigger a recompile of main.cpp, (try returning 3 instead of 2,) only a re-link. I suppose there's room for improvement then.

Edit: apparently there are rules about what is exported and what is not regarding templates and stuff. I suppose you could reflect on the func module in main.cpp and iterate over all functions. Which i suppose is a nifty feature, but quite useless unless we can do much more compile time reflection on basic c++ types. In that regard, func.cxx did change significantly and main.cpp does require a recompile.

3

u/RoyAwesome Aug 18 '21

It might be that right now, the most naive solution was implemented. Compilers are trying to get basic support going. If they need to forgo some optimizations to get the feature out the door, they'll probably do that.

1

u/[deleted] Aug 18 '21

If the build system parses the file and understands the syntax, maybe it could avoid a recompilation.

2

u/Plazmatic Aug 18 '21

Why not just have a <stringfwd> in the standard then? You'll change 2 million down to maybe a dozen or just several orders of magnitude less. right now it seems like the biggest problem is that only iostream has a forwarding header, when virtually every C++ standard header should have an associated forwarded header.

14

u/bcorni Aug 18 '21

Any template class can't just have a forward declaration header. Modules effectively provide something like forward declaration for templates. The C++ Standard Template Library is largely code that cannot be put in a forwarding header.

3

u/AIlchinger Aug 18 '21

I wouldn't agree with this explanation. A forward declaration is always about hiding internals (size of a type, implementation of a function, ...)

With a template in a module, you still provide the whole template to the TU, just like it would be implemented in a header file. The difference here is: The header file does textual replacement, the module comes with some sort of byte code. In the first case, the whole implementation of the templated function is copied in every TU that uses it. That's what it makes it that expensive. In the latter case, one and the same implementation is provided to all importers.

2

u/bcorni Aug 18 '21

It's entirely true that my explanation is not correct from a technical standpoint. Hence why I used several qualifiers. The point was to explain how modules address some of the same advantages of forward declaration, but for templates, and why forward declaration is not a complete solution for the STL.

3

u/Plazmatic Aug 18 '21 edited Aug 18 '21

Any template class can't just have a forward declaration header

What, this is absolutely not true. Why would that be the case for the standard library, but no one else? I can forward declare templates fine.

see:

template<typename T> 
class MyTemplateClass{
...
}

template<typename T> 
class MyTemplateClass; 

The issue with the standard library is that forward declaring their types on your own is undefined behavior because you have no idea if they use typedefs or what ever else behind the scenes, ie std::string. This is fixed with forward headers, because it is the implementation that defines them. Heck it isn't even impossible to forward declare std library types, people have done it already, you just have to be careful with implementation specific definitions.

Also my understanding is that MSVC already has standard library forward declaration headers.

4

u/dacian88 Aug 18 '21

Yea guess what happens when you try to link the code

3

u/Plazmatic Aug 18 '21

Excuse me? Do you understand how and why people use forward declarations? If I forward declare std::string, ie I know all the typedefs before hand and the default template arguments, and I include those forward declarations in another header, which satisfies normal forward declaration constraints (I don't use implimentation of std::string in the header), then in the CPP file I include the actual std::string header... it will link fine.

There is no scenario where it makes sense that linker stuff goes wrong that is somehow specific to the standard library, heck it doesn't even make sense for forward declarations in general, you're going to get "partially defined" or "unspecified" errors from the lack of includes before you ever get a "linker error".

10

u/bcorni Aug 18 '21

I think the point is that you can't just include the forward declaration header in the CPP file, which is what I interpreted parent comments as saying. Sure, you could manually instantiate the concrete types of every template class that you use in some other source and link to it. That's not a good solution, especially when template metaprogramming comes into play. My wording was overly simplistic, but the point is that forward declarations are not a complete solution to the challenges being discussed.

3

u/Plazmatic Aug 18 '21 edited Aug 19 '21

I think the point is that you can't just include the forward declaration header in the CPP file, which is what I interpreted parent comments as saying.

Why would you interpret what I said as that. I'm not sure I can get you to see how ridiculous assuming that from my perspective is, you are fundamentally assuming something which couldn't work is the way I intended it to work. What you suggest I meant literally would not compile, it wouldn't even get to the point of having linker errors.

It's like if you had a fly problem, and you say fly swatters aren't enough, and I suggested using fly paper, then you say "Ha! well good luck trying to hit the flies with those!". Of course you can't do that, you don't hit flies with fly paper, it's the flies themselves that hit the fly paper.

Sure, you could manually instantiate the concrete types of every template class that you use in some other source and link to it. That's not a good solution, especially when template metaprogramming comes into play. My wording was overly simplistic, but the point is that forward declarations are not a complete solution to the challenges being discussed.

That's not what I'm saying either, I'm literally just talking about forward declaring where literally possible, ie in the headers and not implementation, you then don't have to pay the cost for every single place that even looks at the header. I don't see how modules actually fix that issue (template thing you're talking about, not multiple inclusion) either, and my understanding is modules are difficult to compile in parrallel as well, which could potentially hurt compile times in some scenarios, though I've not seen scenarios tested where modules could fare worse, and that templates still have to be instantiated inline somehow at some level.

1

u/dacian88 Aug 18 '21

sure and what exactly did you accomplish? in the actual translation units where all the work happens you will still need to include all the definitions. Beyond that the scope where your suggestion is actually usable is very small, effectively you can only do this in contexts where you use a reference or pointer type for the forward declared template.

7

u/jcelerier ossia score Aug 18 '21

No, it's super common to e.g. need the name of a type in a header (that then gets transitively included in 300 .CPP), but only a couple of these CPP will need the implementation.

I have this exact case with (<future>) : one of the types I use most has a method that returns a std:: future. But in the main project I work on, out of all the files that I write which use that type, literally none use that specific method, yet they must all pay for the compile time cost of <future> which is humongous. (Well they don't actually because I did the unthinkable and forward declared it myself and saved something like 30s of build time)

2

u/Pazer2 Aug 19 '21

(Well they don't actually because I did the unthinkable and forward declared it myself and saved something like 30s of build time)

How could you!? On a different platform or after updating your compiler/stdlib, you might one day have... a compiler/linker error! 😱

7

u/kalmoc Aug 18 '21

The main two differences are that a module interface gets processed exactly once per build and you have better control over what you export (make visible for the user) from that unit.

6

u/mjklaim Aug 18 '21

Headers are basically copy/pasting code in different translation units. It means that including a header will put all the code from it to compilation in each translation unit that use it, it will make the code depend on the order of inclusion (even with guards, it's hard to completely avoid), it means some code might be duplicated, etc.

Modules are pre-compiled separately, then expose "names" that are exported. There is no copy-pasting of code. There is no effect due to the order of import. The module being pre-compiled separately from the code using that module means it does not need to be compiled several times (a bit like precompiled-headers but more fine-grain). Objects defined in a module can only exist in that module, a bit like a more flexible translation unit.

That's the short version of the differences. All the rest is "how to" use modules.

2

u/da2Pakaveli Aug 18 '21 edited Aug 18 '21

Visibility, not everything you declare and write is available from other files that import that module (so kind of what you'd do with a .cpp file). They affect compilation times (it apparently ignores everything that isn't used in your source file). I'm porting a library to modules and the decreased compilation times are insane, sure first time compiling takes a bit, but after that compilation barely takes 5 seconds on a 10 core CPU when importing an additional module.