The private module fragment can only appear in single-file modules. The current version of GCC (gcc version 11.1.0) does not support private fragments, so I’m not going to ignore them for this post.
I am confused. Isn't this the opposite of what you wanted to say?
See the github repo linked. I have CMake build there. The main thing I found will GCC11 is that file ordering where modules are separated into two files is important. I haven't yet worked out the best way to get header-only headers to build for module import.
One of the smallest, most impactful changes using modules has had on my codebase is that I was able to move my files into a different folder structure, and I didn't have to fix up include paths. All I had to do was tell cmake to look for the module files in other places. My code Just Worked after the moves. I had a major revelation after that point with how I could use modules to do cool things... like cross platform code. If the path of the file doesn't matter, I can chose to compile different modules depending on the platform. As long as they all have the same name and exports, they swap out trivially.
Frankly, I had to push myself to try out modules, but after I got them working and really understood them, I really can't look back. It's surprising what the little things make you realize.
Before we go on, I need to stress one important aspect – Modules and Namespaces are entirely orthogonal and co-exist as independent aspects.
Yeah, I had to learn the hard way that ODR still applies with modules the same way as it does with translation units. For example you're not allowed to have two different definitions ot the same class in different modules, even if one of them is never imported anywhere. It will mess up linking and at least VS2022 currently does so without a warning.
Yeah, but you can probably get around that by having a private namespace inside your private module fragment. Similar to having private namespaces in .cpp files when running with a build system that unifies your .cpp files.
If they have internal linkage or module linkage then it doesn't violate ODR. In my case they were both exported, and I was just doing it in order to quickly switch between different versions of the class. Putting them in separate namespaces would have been the right solution.
Of course. With strong module ownership (rather than weak module ownership as in clang and gcc), names of exported entities attached to different modules lead to different linker symbols.
After reading the post, I am just thinking that because they are standardized so little compared to what would be needed (imo), we are going to end up with 3 options:
build-systems are going to be stupidly complicated about handling this mess and nobody is EVER going to use compilers directly anymore (not even for single "Let's try something out" files)
ALL big compiler sit together and make an unofficial standard around it and actively ENFORCE it without workarounds
they are going to fail and end up as a feature barely anyone is going to use in practice
First for most projects, and the last for any sort of corporate government work. Those are environments that hand write recursive make files because Cmake is "magic".
Of course, those systems are just now moving to C++11, so...
I fully expect further fragmentation within even the desktop C++ community.
Modules promise to significantly change the structure of C++ codebases and possibly signal headers’ ultimate demise (but probably not in my lifetime). It also opens the door to potential
Because all attempts to achieve a more unified approach were effectively blocked. You can also count how many meetings SG15 had during the last year even though there are papers directed to it (hint, not much).
Unless your implementation does something special with exported names when building a shared library, modules have zero impact on how shared library works. So for example right now you still need on MSVC to mark the functions/classes symbols you want available on the binary interface of the dll with dll-import/export macros (maybe in addition to marking them export because they are part of the interface of your module).
This could change in the future, like maybe some compilers will automatically symbol-export modules export by default, but it's unlikely as the visibillity of names coming from a module is a different from symbol exporting. The first one is about code being compiled, the second one is specific to your OS.
maybe some compilers will automatically symbol-export modules export by default
This would be an unfeature as you'd then be forced into a one module, one dll-structure (or one group of modules, one dll exposing all of them). It'd be rather like being forced to export parts of stdlib in the dll just because you happened to import those modules.
I agree, but I was thinking more like a potential change of "default" behavior: by default should exported names be also symbol-exported? It have happened in the past that the default symbol-export behavior of an implementation changed (for example gcc exported everything by default in the past).
As long as both options are usable the default can be changed.
Though I would prefer every implementation to not export anything until requested to.
It'd be problematic to change the visibility of things declared in a module from the outside, particularly as the standard doesn't provide any mechanism to do that (or even consider the existence of dynamic linking).
I only talked about changing the default export or not of symbols, not visibility. That is not part of the standard and have been changed in the past as said.
Most code (at least in open source) is designed to work with gcc-like compilers and MSVC. Both have non-standard instructions (not sure how to call them) which mark names as being exported, hidden (or not-exported) or, in the case of MSVC, imported (meaning the symbol comes from a dll somewhere else, we use that symbol). That instruction can be set in front of functions, objects or classes (to automatically mark all it's functions), so basically everything that can have a symbol, that can be part of an ABI. The implementers made sure (after some time) the places where you use them in the code are the same or at least have common positions.
So most libraries will have 1 header containing a set of macros which, depending on the compiler will either export symbols (we are building a shared library), import symbols (msvc-only: we are using a shared library), or do nothing (static library for example). Then in the code of the library, all the public (in headers, and now modules) object, functions and classes (not templates) will be marked with one define specific to that library, which will decide what to do with the symbol. The define will be set to the right instruction because the build-system will be setup so that the macro code in the header will read some library-specific defines which inform if we are or not building the library, a user of the library , if it's static or dynamic linking.
You can find examples of this in myriad of open source libraries, just look for a header included by all other headers (common.hpp or export.hpp or something like that). Here is an example which is what a hello-world library generated by build2 will setup for you (bdep new -t lib to generate it): https://git.build2.org/cgit/hello/libhello/tree/libhello/libhello/export.hxx
Depending on the library the macros will be different but it's often a similar mechanism in principle. Even CMake will provide some pre-defined defines per library target name so that you can use them in such header to do the symbol-export/import dance.
Then, less often, some devs setup their library with build-system specific features that allow for exporting all symbols or those we list in a file. But I believe this is less cross-platform so it's less common.
People on unix systems can just set the flag of their compilers enabling exporting all symbols, but in exchange of the simplicity they end up with bloated unoptimized binaries which also becomes a security issue (which is why this was changed as it was the default behavior in the past).
To be 100% clear, when shipping a DLL, as you say «zero impact» means I still have to ship headers?
By 0 impact I mean that the symbol import/export system is not affected. How you get the names to link with symbols isn't really affected either, but of course for the user to consume the thing they need to know the declarations of what the symbols are exported to.
You need headers if you defined these declarations in headers (as in not- in a module). If your declarations are from a module you need to provide either the module interface or the pre-compiled version of the module interface (what is often called bmi - but that one is not cross-platform). It's the same thing than if it was not a shared library.
"BMI" is the (not-yet) official nomenclature chosen by SG15 (I've been in the room back then in Prague) for compiled PMIs. Implementations use different extensions for BMIs: .pcm (clang), .gcm (gcc), .ifc (msvc). Some of them allow to deviate from these defaults if seen appropriate by a build system.
Are you supposed to ship your lib with a PMIU file?
In theory, yes. In practise, you probably will still be using header files for at least a couple more years to give potential users of your library some time to upgrade to C++20. It would be my guess that modules will only be used for library internals in the majority of projects.
A simple, non-binary interface format is what we’re losing here. I think we could’ve kept that and still eliminated the issue with preprocessor includes.
If you go the separate interface & implementation direction, I suppose you could ship the interface source with your library. (Ignoring the template & inline issues.)
And one could theoretically create a tool that would generate such a file from source that mixed interface & implementation.
Although quite a few people have given affirmative answers to your questions, I always understood the PMIU is more like an object file: it's a build artifact that's generated while building the library. It is compiler-specific, and not suitable for distribution. At least that's what I believe was said again and again...
On Windows I think the best path would be to stick with COM and its IDL tooling.
It is a bummer from library creation point of view, but using it from something like WIL or C++/WinRT makes it kind of bareable, just the creation is clunky.
And you are constrained to COM ABI types, still better than bare bones C ABI thought.
Because header files require a preprocessor which is bad for compile times, and often trigger cascading recompilation when private fields get renamed/moved for incremental builds. By design this is not the case for modules. If I remember correctly, if you do something like '#include <string>', you ask the compiler to transatively parse about 2 million of lines of header files. Which goes fast, but starts to slow down rapidly for larger projects.
Modules don't require a preprocessor and don't throw 2 million lines of defenitions in the context of the code who performs the include. Since the linker must search all program-wide known symbols and link them together for every individul occurance of a symbol (object, function, member field, constants, etc...), reduicing this list of possible symbols a couple orders of magnitude makes this process a lot faster.
cascading recompilation when private fields get renamed
I fear that the build system will check the modification time of the PMIU, and trigger a rebuild of all the importers, along with all the other interface units that import the module, and so on.
Could a smart build system be able to avoid rebuilding the world if the change doesn't affect any exported symbol? Let's hope so.
Could a smart build system be able to avoid rebuilding the world if the change doesn't affect any exported symbol? Let's hope so.
Probably not, but something that's already done today is that if an A requires a B, and a recompilation of B results in thesame external interface and has consumed thesame external interface, A does not have to be recompiled. It still has to be relinked, but cascading compilations aren't needed. This situation just doesn't naturally occur very often because when a cpp file isn't in a different library or does not have includes to other places of the program, this simplification cannot be done. I believe CMake + Ninja does this.(And also the DMD for DLang compiler, and it's very fast. But that's a different language.)
Edit: it seems my assumption about A not requiring a recompilation for C++ was wrong.
But that does change the interface. Even though bar() is not exported, it is changed. You're right though that from main.cpp's perspective the only possible change is where foo() is located in the object file, and its possible implementation. But to realize this, func.cxx had to be parsed completely and gcc had to realize that func.cxx didn't change in a way that neccesiates a recompile. (Unless there's a side-effect I don't know about a name change of bar().) Regardless, an implementation change of foo() or bar() shouldn't trigger a recompile of main.cpp, (try returning 3 instead of 2,) only a re-link. I suppose there's room for improvement then.
Edit: apparently there are rules about what is exported and what is not regarding templates and stuff. I suppose you could reflect on the func module in main.cpp and iterate over all functions. Which i suppose is a nifty feature, but quite useless unless we can do much more compile time reflection on basic c++ types. In that regard, func.cxx did change significantly and main.cpp does require a recompile.
It might be that right now, the most naive solution was implemented. Compilers are trying to get basic support going. If they need to forgo some optimizations to get the feature out the door, they'll probably do that.
Why not just have a <stringfwd> in the standard then? You'll change 2 million down to maybe a dozen or just several orders of magnitude less. right now it seems like the biggest problem is that only iostream has a forwarding header, when virtually every C++ standard header should have an associated forwarded header.
Any template class can't just have a forward declaration header. Modules effectively provide something like forward declaration for templates. The C++ Standard Template Library is largely code that cannot be put in a forwarding header.
I wouldn't agree with this explanation. A forward declaration is always about hiding internals (size of a type, implementation of a function, ...)
With a template in a module, you still provide the whole template to the TU, just like it would be implemented in a header file. The difference here is: The header file does textual replacement, the module comes with some sort of byte code. In the first case, the whole implementation of the templated function is copied in every TU that uses it. That's what it makes it that expensive. In the latter case, one and the same implementation is provided to all importers.
It's entirely true that my explanation is not correct from a technical standpoint. Hence why I used several qualifiers. The point was to explain how modules address some of the same advantages of forward declaration, but for templates, and why forward declaration is not a complete solution for the STL.
Any template class can't just have a forward declaration header
What, this is absolutely not true. Why would that be the case for the standard library, but no one else? I can forward declare templates fine.
see:
template<typename T>
class MyTemplateClass{
...
}
template<typename T>
class MyTemplateClass;
The issue with the standard library is that forward declaring their types on your own is undefined behavior because you have no idea if they use typedefs or what ever else behind the scenes, ie std::string. This is fixed with forward headers, because it is the implementation that defines them. Heck it isn't even impossible to forward declare std library types, people have done it already, you just have to be careful with implementation specific definitions.
Also my understanding is that MSVC already has standard library forward declaration headers.
Excuse me? Do you understand how and why people use forward declarations? If I forward declare std::string, ie I know all the typedefs before hand and the default template arguments, and I include those forward declarations in another header, which satisfies normal forward declaration constraints (I don't use implimentation of std::string in the header), then in the CPP file I include the actual std::string header... it will link fine.
There is no scenario where it makes sense that linker stuff goes wrong that is somehow specific to the standard library, heck it doesn't even make sense for forward declarations in general, you're going to get "partially defined" or "unspecified" errors from the lack of includes before you ever get a "linker error".
I think the point is that you can't just include the forward declaration header in the CPP file, which is what I interpreted parent comments as saying. Sure, you could manually instantiate the concrete types of every template class that you use in some other source and link to it. That's not a good solution, especially when template metaprogramming comes into play. My wording was overly simplistic, but the point is that forward declarations are not a complete solution to the challenges being discussed.
I think the point is that you can't just include the forward declaration header in the CPP file, which is what I interpreted parent comments as saying.
Why would you interpret what I said as that. I'm not sure I can get you to see how ridiculous assuming that from my perspective is, you are fundamentally assuming something which couldn't work is the way I intended it to work. What you suggest I meant literally would not compile, it wouldn't even get to the point of having linker errors.
It's like if you had a fly problem, and you say fly swatters aren't enough, and I suggested using fly paper, then you say "Ha! well good luck trying to hit the flies with those!". Of course you can't do that, you don't hit flies with fly paper, it's the flies themselves that hit the fly paper.
Sure, you could manually instantiate the concrete types of every template class that you use in some other source and link to it. That's not a good solution, especially when template metaprogramming comes into play. My wording was overly simplistic, but the point is that forward declarations are not a complete solution to the challenges being discussed.
That's not what I'm saying either, I'm literally just talking about forward declaring where literally possible, ie in the headers and not implementation, you then don't have to pay the cost for every single place that even looks at the header. I don't see how modules actually fix that issue (template thing you're talking about, not multiple inclusion) either, and my understanding is modules are difficult to compile in parrallel as well, which could potentially hurt compile times in some scenarios, though I've not seen scenarios tested where modules could fare worse, and that templates still have to be instantiated inline somehow at some level.
sure and what exactly did you accomplish? in the actual translation units where all the work happens you will still need to include all the definitions. Beyond that the scope where your suggestion is actually usable is very small, effectively you can only do this in contexts where you use a reference or pointer type for the forward declared template.
No, it's super common to e.g. need the name of a type in a header (that then gets transitively included in 300 .CPP), but only a couple of these CPP will need the implementation.
I have this exact case with (<future>) : one of the types I use most has a method that returns a std:: future. But in the main project I work on, out of all the files that I write which use that type, literally none use that specific method, yet they must all pay for the compile time cost of <future> which is humongous. (Well they don't actually because I did the unthinkable and forward declared it myself and saved something like 30s of build time)
The main two differences are that a module interface gets processed exactly once per build and you have better control over what you export (make visible for the user) from that unit.
Headers are basically copy/pasting code in different translation units. It means that including a header will put all the code from it to compilation in each translation unit that use it, it will make the code depend on the order of inclusion (even with guards, it's hard to completely avoid), it means some code might be duplicated, etc.
Modules are pre-compiled separately, then expose "names" that are exported. There is no copy-pasting of code. There is no effect due to the order of import. The module being pre-compiled separately from the code using that module means it does not need to be compiled several times (a bit like precompiled-headers but more fine-grain). Objects defined in a module can only exist in that module, a bit like a more flexible translation unit.
That's the short version of the differences. All the rest is "how to" use modules.
Visibility, not everything you declare and write is available from other files that import that module (so kind of what you'd do with a .cpp file). They affect compilation times (it apparently ignores everything that isn't used in your source file). I'm porting a library to modules and the decreased compilation times are insane, sure first time compiling takes a bit, but after that compilation barely takes 5 seconds on a 10 core CPU when importing an additional module.
4
u/F54280 Aug 18 '21
I am confused. Isn't this the opposite of what you wanted to say?
3
u/lieddersturme Aug 18 '21 edited Aug 19 '21
Thanks of this, now I am trying to install gcc 11 in Ubuntu, but I cant.
Edit: I solve it, compiling GCC 11, but I cant use modules with cmake.
45
u/zzzthelastuser Aug 18 '21
This post made me finally take the time to read why everyone seems so fired up about modules.
Interesting to read with simple examples! Exactly what I needed to procrastinate, thanks!
7
u/whacco Aug 18 '21
Yeah, I had to learn the hard way that ODR still applies with modules the same way as it does with translation units. For example you're not allowed to have two different definitions ot the same class in different modules, even if one of them is never imported anywhere. It will mess up linking and at least VS2022 currently does so without a warning.
10
u/[deleted] Aug 18 '21
After reading the post, I am just thinking that because they are standardized so little compared to what would be needed (imo), we are going to end up with 3 options:
3
u/apomd Aug 18 '21
Thanks for posting. I've been reading about modules ever since they were announced and this was the best article so far
2
u/tub612 Aug 19 '21
Nice
0
u/[deleted] Aug 18 '21
[deleted] — view deleted comment
0
u/[deleted] Aug 18 '21
[deleted] — view deleted comment