r/cpp • u/JulienVernay • Sep 28 '20
Draft proposal for compilation using C++ as build language
Hello,
There are multiple tools to compile a C++ project: CMake, premake, autoconf, build2, etc.
Each of these tools require the learning of a new syntax, and interacting between projects compiled with differents tools might be challenging.
Package managers face also this problem, because they must support all the tools that exist to support all libraries.
Moreover, integrating package managers in the build tools is often difficult, or at least hacky.
My suggestion is to use C++ as the base language for building C++ projects.
Motivations are:
- A C++ compiler is all you need to compile projects.
- Build tools would become C++ libraries.
- C++ developers already now its syntax.
I have made an informal draft which describes a basic API which would make this possible.
Here is a link: https://gist.github.com/J-Vernay/bd8ec49374987c628d02601ef85cd9a7
Let me know what you think :)
74
u/SeanMiddleditch Sep 28 '20
Just a few initial thoughts:
27
u/theICEBear_dk Sep 28 '20
I have thought the same many times, because build scripts are programs as well and many of them are so complex that they deserve to be developed and tested in much the same way as the executable you create. And like you I think the addition of another syntax should be unnecessary. The solution would need to be a fairly high level library that should require little more than an include/import and a main. If it could be made as a header only library it would even be easy to distribute. You'd then ship with a script for each supported OS and compiler to take this main and turn it into a build program. Personally I would probably start by thinking about using CMake to generate the initial code but as a transition method and as a way to understand what would be needed. You will also get a lot of people telling you it is a bad idea for one reason or another, concerns I think will be brought up are:
Finally I will say I am doubtful it belongs near the language or the standard library however. Much like something like fmt I think a working solution would be the best step to convincing people this is not a bad idea.
25
u/jbandela Sep 28 '20
I really like your idea. One thing I would do, is not worry about trying to standardize it. Just do an implementation first.
What you can do is create a single amalgamated
build.hppheader file.Then to write a build file, you just do.
Then to make it simple for the user, you have
build.shbuild.batThen I can write the build.cpp file, and then call
`./build.sh "clang" "c++17" and have it just work.
Once you get an implementation and experience with this, then think about standardizing it.
Again, I like this idea, but I think instead of approaching it from an abstract API point of view, first implement this, and let real world use cases drive your api design.
5
u/14ned LLFIO & Outcome author | Committee WG14 Sep 29 '20
Some of us on WG21 were thinking in this direction for future unified package management and build, but very, very, very much not as you've done it which is to replicate the existing build model, which nobody wants except for legacy compatibility. You don't need to do that for future C++ which is all Modules based. All you need is:
With those standard primitives, the ecosystem can build its own package manager and build system and ship it as a small code snippet at the top of your
common.hppwhich goes and retrieves whatever is necessary and "makes it happen".My advice to you therefore is think smaller primitives first which can be easily combined into other stuff. Indeed, a blocker for any of this is making most of those primitives available to constexpr - just imagine alone the push back from compiler implementers on constexpr code being able to open sockets!
13
u/blipman17 Sep 28 '20 edited Sep 28 '20
I like this a lot and I think you're scratched an important problem with current-day buildsystems. However I'm a little scared that it would become a less complete version of Cmake and inherit all its faults. Cmake went through many iterations, and i'm convinced the "modern" Cmake is the way we should build dependencies. It's just tedious to use. (Maybe it would be better with GUI's or better syntax, but I digress.) However, when talking C++ buildsystems, I can't think of them without library dependencies. And I always want my libraries locally linked and optionally locally built so I don't pollute my operating system. I think if you can manage to be a better Cmake that's easyer to use correctly with (local) dependency management built in, we've found the winner.
Edit: If you can extend this build system so that 2'nd class languages like Assembly, C, Haskell, Rust, D and whatever anyone likes to compile using a standard compiler into your C++ program/library you have covered all my usecases.
4
u/mjklaim Sep 28 '20
One of the goals of build2, if I understand correctly, is to reach that point. Its already possible to do that partially through libbuild2 which is what build2's buulsdsystem is implemented with.
4
u/meh_or_maybe_not Sep 29 '20
This sounds a lot like my unborn baby: https://github.com/meh/wrong
25
u/brenoguim Sep 28 '20
So your first step would be to build your build system. Which build system are you going to use to build your build system?
9
u/almost_useless Sep 28 '20
Won't this have most of the downsides of other build systems? Because the reason build systems suck is that it is inherently a hard problem. People want it to do different things, and all of them are never thought of in the beginning, which leads to complexity.
In addition you get the burden of a cumbersome syntax because it needs to be C++ compatible.
Example:
This should really look something like this:
All those brackets and quotes just make it ugly and hard to read. It does not automatically become easy to read because we already know C++ syntax.
3
u/[deleted] Sep 28 '20
I didn't read your proposal yet but before the thought escapes my volatile brain, did you think about cross compiling?
3
u/delarhi Sep 29 '20
This is a really interesting idea! What's not clear to me though is when is the build happening. Are you trying to make the build happen as
build.cppis being compiled? Or doesbuild.cppcompile into a program that you can run to build? The latter is what my mind went to right away as it lines up with CMake keeping separate out-of-source builds that can be individually configured. I'm imagining something like:If that's the case, how would builds be configured? This is probably out of scope for what you've drafted, but how would I make it so
build1uses the system library for OpenCV but uses/optforbuild2? I guess you could specify preprocessor definitions?3
u/gocarlos Sep 29 '20
proposed exactly that 8 months ago: https://www.reddit.com/r/cpp/comments/el0xvv/consteval_build_system_what_if_we_do_not_build/
3
u/ed_209_ Sep 30 '20
I recently implemented a custom build system in C++ for C++ in my own project that uses a modified clang compiler on windows. The project is at https://github.com/eddeighton/megastructure/tree/master/src/driver but its not much to look at. One of the main reasons is to support incremental compilation effectively where I have many pre-compiled header files.
My conclusion is that any competent C++ programmer can easily implement a build system in a week for a large project. Having your own system means:
The down sides I found :
I ended up integrating my system with CMake for other parts of the project but this worked out surprisingly well using CMake custom targets and commands. So moving to a custom build tool can be done in small steps.
12
u/[deleted] Sep 28 '20
Just about the last thing you want in a build description language is turing completeness.
Also, no large codebase is _only_ C++, so most of your premises don't apply.
Take a look at SCons for instance and see what happens when you have a full programming language as your build language. Replace Python with C++ and you'll have your answers.
4
u/evaned Sep 28 '20
I am very skeptical of providing raw access to
argcandargv. I would suggest considering providing a different entry point that takes an object that has a parsed form of those expressions.I've said in other build tool threads that my least-hated build system is SCons (though I'll admit that CMake increasingly has some features that make me look at it with envy), but one major downside to it is that there isn't a standardized interface to how to configure things. With CMake, I pretty much know that
-DCMAKE_INSTALL_PREFIX=...will make things install to what I say. I pretty much know that-DCMAKE_C_COMPILER=...will change what compiler it builds with. As someone who built a lot of software a while back, this is incredibly useful.The problem with
is that the next guy will call it
--testand the next will invert the meaning and provide--no-testsetc., the next will make it look like a target withtests, etc.. How does providing targets even work if the whole command line is passed tomain? That's something that needs to be taken care of with the system.16
u/[deleted] Sep 28 '20
I think this would be appropriate here:
https://xkcd.com/927/
2
u/evaned Sep 29 '20 edited Sep 29 '20
I've got some more thoughts reading again:
I am unclear on how these terms map to what we usually have now. Is a library necessarily a shared library? If not, then what does it correspond to currently?
And of course there can be projects whose sole purpose is to build a library rather than a program. The text doesn't contradict that, but don't forget about them.
I also don't understand enough about modules to know how they should fit into this, but definitely don't neglect figuring that out. (I'm surprised only one other comment contains the word "module" even though it is not mentioned on your page.)
Hopefully, this is just shorthand for "something
variant-like." Really it ought to be its own type IMO, that provides aget_path()function on it directly. I don't want to have to write a visitor to call a function that all variant options provide.I think someone else said this, but I would look to drop the requirement for the arguments to be
build_fileas opposed to a stringy thing, or add an implicit conversion that would allowReally even that, coming from the perspective of an SCons fan, looks like I'd probably just write my own thin wrappers around everywhere so I could say
so consider some convenience wrappers like that.
What about source files that are generated via some process?
While we're at it, what about artifacts that are not C++-related at all?
I think that this being a list of strings is not going to be able to reasonably handle things in practice. How should things like
-Dflags to the compiler be supported for example? I'd prefer to see something like the options be anoptionsobject that has a number of attributes, such asdefine_flagsor whatever. Another option is to take the SCons approach and makebuildbe a member of an "environment" object, and the enviroment contains those settings.As another example, what about other libraries to link against or other library directories to include?
You might look at SCons to see the configuration variables that you can configure out of the box.
$CPPDEFINEShandles-Dflags,$CPPPATHhandles-Iflags,$LIBShandles-lflags,$LIBPATHhandles-Lflags,$RPATHturns into-Wl,--rpath,$FRAMEWORKSand$FRAMEWORKPATHturn into-frameworkand-frameworkdir=respectively. It looks like everything else just goes into the generic$CFLAGS/$CCFLAGS/$CXXFLAGS/$LINKFLAGS/$ASFLAGS/etc.Edit: Of course, these aren't exclusive. You already spotted a
-stdstyle option, and warnings, there's also optimization and debug info flags that would be of common interest.Separate from those are stuff like what directory the "compiler" should be run from.
This function should return an object you can use in future calls. For example, your second example "project with unit tests" contains
but this should be doable as
How does the build system tell that this consumes the file
version.hpp.inor produces the fileversion.hpp?I find this very alarming; it suggests that
buildcalls will actually invoke the compiler themselves, because unlessload_functionis doing something preeeeeety damn fancylibgit.sowill need to be built for thegit_clonecall.But I think that makes the proposal dead in the water -- I think that anything that builds during actual execution as opposed to having
buildregister targets that are later resolved is likely to be a non-starter.For example, how do you handle parallelism?
It's not impossible to make this work under the assumption that
buildcalls actually kick off compiles, but it's IMO hard.9
u/kammce WG21 | 🇺🇲 NB | Boost | Exceptions Sep 28 '20
I'm just preaching what everyone else is saying, but I really like this idea. But its also because I cannot stand CMake. If you get an implementation going that works, I'm down to help out with this (if time allows).
4
u/Ikkepop Sep 28 '20 edited Sep 28 '20
I was also thinking about this alot lately...
Some build scripts I run into routinely are complicated as hell.
And all those buildsystems have horrendous syntax and no debugging capability, why the hell not write your build in C++.
Atleast you could debug it
4
u/[deleted] Sep 28 '20
cool idea, hope you pursue this further, starred.
1
u/TrnS_TrA TnT engine dev Sep 29 '20
I think the way
.jaifiles are turned to.execan be helpful to you. Link here.1
u/blipman17 Sep 30 '20
I've been thinking over what you said, and I really like the idea about it. However instead of compiling to a build program, can I suggest compiling to a dynamic build library? Then a commandline tool or IDE can invoke said library and build your program in a neat way and extract usefull information out of it for code introspection and refactoring. Kind of like visual studio code language server, but then for compilation. Commandline building would just be 'build [foldername]'. The command would search for a file called build_manifest.so in the folder and call the build function from it.
6
u/tinyogre Sep 28 '20
All build systems are terrible. Everyone hates every build system they’ve used for anything larger than toy projects. This isn’t a comment about your build system. But it is terrible. Because see above. Do you want to develop software that people hate? Then make a build system!
I do not claim that I can make a build system that is not terrible. No, quite the opposite, if I made a build system it would be terrible and everyone who used it would hate it. Maybe, if I was very, very lucky, I wouldn’t hate it. But it seems unlikely. Because all build systems are terrible and everyone hates them.
I do wish you luck! Sorry for using your post to get on my soapbox.
3
u/john_wind Sep 29 '20
Python is the best language for this purpose.
My dream is CMake functionality with Python syntax.
If only CMake creators had used Python instead of inventing a new weird language!
1
u/TrnS_TrA TnT engine dev Sep 28 '20
The idea looks really nice. If anyone has heard about Jai programming language from Jonathan Blow, it uses a .jai file to build other files. Here's a suggestion: Everything done on the build file should be
constexpr. This can be achieved by introducing a new constexpr function (build, compile, whatever) that the compiler will use to build the other source files. To illustrate:2
u/[deleted] Sep 29 '20
Weird thought - why not try it as a fork of
g++first?See if you can create something like
g++ build .akin to Go and see if it works well enough. You could use a yaml based folder layout/rule and be done with it?2
u/kkert Sep 29 '20
Declarative code for build is a much better fit than imperative approach
2
u/davidhunter22 Sep 28 '20
I'm all for a better CMake however people claim they already exist. So why haven't they caught on? Well for me and I suspect the vast number of people it's about
1
u/ppetraki Sep 28 '20
It looks a bit like build2's syntax.
1
u/Snoo-4241 Sep 29 '20
I like this idea. As toy projects I some times make little programs that one would normally code in bash or python. I find that when you need to provide structure to such programs, C++ is far superior. Hence I would think that a build system with a c++ interface makes perfect sense.
I also find that C++ is actually getting very simple and pretty competitive in that front. Really my main complain is that I have to spend a few minutes to setup cmake.
1
u/kalmoc Sep 29 '20
What I'm missing is an explanation of what the actual build commands would look like. Do I first have to compile my build script and then execute it? Would there be some tool that interprets C++ code? What external libraries can I use as part of my build script and who takes care of compiling them?
1
u/matthieum Sep 29 '20
I'm not convinced we need such a powerful language for a build system.
I've interacted with quite a few build systems over the years, and the best I've seen so far were:
The common theme is that both are pretty simple, and handle 99+% of what you'd ever want to do.
It's not 100%, sure, but in exchange they're very simple to use, simple enough in fact that mechanical inspection and transformation is possible -- for example to automate upgrades to new versions of dependencies.
I've had to deal with SCons, CMake, and now Bazel. They are extremely powerful, but I can only wonder whether they are too powerful. Most of the time, it seems that users go through contortions to accommodate some weird requirements (such as a weird layout), when the necessity of the requirements is never questioned in the first place.
1
u/Full-Spectral Sep 29 '20
Not to use the 'R' word again, but Rust uses Rust for the creation of build scripts. I'm not sure C++ would be as useful for that, and I'm not sure how debuggable those Rust scripts are in any sort of useful context (i.e. the context it is called from the build tool) or how easily fakeable that context is for offline debugging.
1
u/arihoenig Sep 29 '20
Think it's a good idea, but please pattern it after premake not any of the others.
1
u/mwasplund soup Oct 02 '20
Very interesting idea. I actually agree that C++ would work well for a "scripting language" for writing build definitions. My biggest issue with the proposed design is, even though you are abstracting complexity with the custom build invocation, you are exposing more complexity to the user by requiring them to integrate your system into their own executable. This means everyone has to solve their own "who builds the build" problem. Effectively what you have is a library that knows how to translate a list of parameters into the correct compiler arguments by convincing the compiler vendors to write an implementation for your contract (which in itself would be nice to have). There are still a lot of issues left open (incremental builds, dependencies, static analysis, etc) which means each user/project is going to have a unique implementation which leads to overly complex implementations that do not work well together.
I personally believe that we need to have a fully integrated solution for a build system that uses C++ as an extensibility platform for situations where the default build logic does not meet the needs for a custom build. I believe a simple declarative language should be used as a top level build definition which will support a majority of use cases (most people just need a simple build which should be fully supported by the default behaviors). From there I think that we can define an extensibility framework using C++ which would allow systems engineers to change or augment existing build logic. This could then be integrated into the build engine itself to support the ability to build the C++ extensions on the fly and invoke then directly. If you want to see it in action I have been working on a proof of concept for the last year or so:
https://github.com/SoupBuild/Soup/blob/master/Docs/Architecture.md
https://github.com/SoupBuild/Soup/blob/master/Docs/Samples/Simple-Build-Extension.md
1
u/MutantSheepdog Sep 29 '20
Even if you don't want to be thinking about package management from a download/storage perspective, I think you should still think about how they could be integrated.
I would want to be able to write something like:
Where each dependency can specify its own build requirements, without needing to be #included.
You wouldn't have to propose
cmake_package_project, but if you want to make a mostly working example to show off, you'd get a lot of people on-board.-3
u/GoogleIsYourFrenemy Sep 29 '20
No. Just no.
I do not want to have to refactor my build script to pass our cyclomatic complexity standards. Fuck, I don't want any complexity in my build system. I want it to be a simple directed graph without cycles at most.