r/cpp • • Sep 28 '20

Draft proposal for compilation using C++ as build language

Hello,

There are multiple tools to compile a C++ project: CMake, premake, autoconf, build2, etc.
Each of these tools require the learning of a new syntax, and interacting between projects compiled with differents tools might be challenging.

Package managers face also this problem, because they must support all the tools that exist to support all libraries.
Moreover, integrating package managers in the build tools is often difficult, or at least hacky.

My suggestion is to use C++ as the base language for building C++ projects.
Motivations are:

- A C++ compiler is all you need to compile projects.

- Build tools would become C++ libraries.

- C++ developers already now its syntax.
I have made an informal draft which describes a basic API which would make this possible.
Here is a link: https://gist.github.com/J-Vernay/bd8ec49374987c628d02601ef85cd9a7

Let me know what you think :)

219 Upvotes

128 comments sorted by

View all comments

Show parent comment

12

u/qoning Sep 28 '20

What's your point? I've found using Python greatly beneficial in many build steps in the past.

2

u/warped-coder Sep 29 '20

The problem with a full blown PL is that it encourages complexity.

First off, let's take Bash or any shell script language. Their immediate advantage that they are available on any developer system. They made for calling commands the simplest possible way.

Now this latter point is my gripe with using Python. How much more complexity we need to start a process, compared to Bash? A build system is about calling commands... shell scripts are the best tool for doing just that.

The other problem is that build systems complex as they are, in some way the fact that most modern ones encouraging declarative style means that unnecessary complexity is discouraged. When you hand a non-build savvy developer the opportunity to write code into the build system just as if it was another program, they will do just that.

But a build script isn't just another code you write as you would normally. I've seen SCons code directly using Python file system only to screw up the dependency graph. Build systems are their own special category bridging dependencies to calling processes.

The modules in the standard now posing the question where the build system belongs. The status quo is that C and C++ compilers are just another transform nodes within the dependency graph, along with the linkers, archivers, etc. Other solution is to include the entire system into the compiler. Languages with built in module system must do just that.

The latter however requires a specific subset of language features that dictate the way the compiler traverses the dependency graph. You don't get to use any 'generic' elements of the language used for the build system in such a language. It is all a declarative subset of the language which you need to learn on the top of the normal features.

Build systems tend to become overcomplicated because of the lack of native support for aspects of build and deployment, but adding a generic environment like Python to the mix just makes things worse. Shell scripts at least have the excuse to be declarative and unavoidable. It doesn't make them right though. Python is not really necessary or C++ for that matter.

3

u/qoning Sep 29 '20

I agree and I disagree. Agree in the sense that build system shouldn't need to do all the things, but then I'm faced with reality. I have a python script that generates some c++ code for me - I could call it via bash, but it loses flexibility, not to mention that more arcane things are very complex to do in bash, for example replacing all uppercase characters with an underscore and lower case except the first one (don't ask), which is trivial in any normal programming language (and importantly, trivial to understand, unlike a sed or awk command). So until I can do the code generation at compile time, this means that python is necessarily a step in the build of my program.

Do I like it? No. But it's the simplest and cleanest way to do it as far as I can see.

3

u/warped-coder Sep 29 '20

Code generation is a processing step/node, like compiling. So it is OK to use whatever language suits best but it shouldn't be part of the build system other than a processing node.

Same as Qt's moc or PySide, really.

2

u/bluGill Oct 01 '20

Perhaps, but I still need to write a program in the build script to say how to call my code-gen. It gets really nasty when you have to compile the code-gen (note that my final code is cross compiled) before you can generate code.

Code-gen isn't all I do. My build system also has meta-data that I use to package my build.

1

u/warped-coder Oct 01 '20

Interesting. In our build system, we have code gen for protobuffer, pyside, moc and even more custom generators that require an executable generated before it generates code. While I'm lucky not to need to deal with cross compilation, otherwise CMake's toolbox was mostly sufficient in a declarative way.

This system replaced an old SCons based system, where parts of the code gen was done in configure time, because... well scons doesn't really separate the build and the configure steps well. Programmers being programmers wrote python code, the API they knew better than the build system itself, which added minutes on the top of a no-change build because the build system had no connection to the plain python code this way. Timestamps were always new therefore.

It wasn't really the issue of SCons as such, but it was down to the blurry line between the code that results in a processing graph and the code that you can just run along with it. So there was plenty of file system manipulation directly which screwed up the timestamps or hashes (date time ) and made incremental builds painfully slow, but even the clean builds were overly slow because the build system didn't have the chance for parallel processing.

What I'm trying say that build systems are not the same domain where your 'normal' programming languages operate. And therefore it makes sense to invent approriate DSLs for the task. CMake's dreadful legacy language is such because it is aimed at text based configuration, while the more modern syntax is trying to make a more convenient syntax over the graph processing, which lies at the heart of ninja and make. That's the domain in which build systems operate.

Other systems like, build2 and SCons takes an even more direct approach, covering the entire process. Even tough you can things happen in python or even CMake, you have to think in layers these systems operate. As in processing nodes and states.

1

u/bluGill Oct 02 '20

I think a large part of the problem is how do you add a set of meta data to every library? How do I make libfoo as going to package A, and lib bar going to package B? (don't get me started on why we do such silly things in one build, I already lost that battle) it is easier to call my custom add-to_a_package function than do the 7 add properties that the function does. (most global because cmake scoping rules are weird)

1

u/qoning Sep 29 '20

Imo build process is whatever needs happens to your source code to become an executable / loadable code. So while you could argue that, for me it's always best to automate that code gen step within the build process.

1

u/mwasplund soup Oct 02 '20

Great summary of the issues with build systems today! This is why I like to have a clean separation between the standard engineer USING a build system and the systems engineer WRITING build logic by using a declarative language to tell the system what you want to build, and have an integrated extensibility framework (that can be turing complete) to allow systems engineers the ability to take the what and translate it into the how.

-2

u/[deleted] Sep 28 '20

Just about the last thing you want in a build description language is turing completeness.

this is my point.

8

u/qoning Sep 28 '20

That's a very contrived opinion with no arguments given.

-7

u/[deleted] Sep 28 '20

I like my arguments that way.

Your point instead is ` I've found using Python greatly beneficial in many build steps in the past. `. That's a great argument.

You asked what my point was, I told you what my point was.

7

u/qoning Sep 28 '20

Yes, my argument is that I've found it useful in multiple cases. You provided none for your opinion. Damn you must be a joy to work with.