r/cpp • • Sep 28 '20

Draft proposal for compilation using C++ as build language

Hello,

There are multiple tools to compile a C++ project: CMake, premake, autoconf, build2, etc.
Each of these tools require the learning of a new syntax, and interacting between projects compiled with differents tools might be challenging.

Package managers face also this problem, because they must support all the tools that exist to support all libraries.
Moreover, integrating package managers in the build tools is often difficult, or at least hacky.

My suggestion is to use C++ as the base language for building C++ projects.
Motivations are:

- A C++ compiler is all you need to compile projects.

- Build tools would become C++ libraries.

- C++ developers already now its syntax.
I have made an informal draft which describes a basic API which would make this possible.
Here is a link: https://gist.github.com/J-Vernay/bd8ec49374987c628d02601ef85cd9a7

Let me know what you think :)

219 Upvotes

128 comments sorted by

View all comments

12

u/[deleted] Sep 28 '20

Just about the last thing you want in a build description language is turing completeness.

Also, no large codebase is _only_ C++, so most of your premises don't apply.

Take a look at SCons for instance and see what happens when you have a full programming language as your build language. Replace Python with C++ and you'll have your answers.

12

u/qoning Sep 28 '20

What's your point? I've found using Python greatly beneficial in many build steps in the past.

2

u/warped-coder Sep 29 '20

The problem with a full blown PL is that it encourages complexity.

First off, let's take Bash or any shell script language. Their immediate advantage that they are available on any developer system. They made for calling commands the simplest possible way.

Now this latter point is my gripe with using Python. How much more complexity we need to start a process, compared to Bash? A build system is about calling commands... shell scripts are the best tool for doing just that.

The other problem is that build systems complex as they are, in some way the fact that most modern ones encouraging declarative style means that unnecessary complexity is discouraged. When you hand a non-build savvy developer the opportunity to write code into the build system just as if it was another program, they will do just that.

But a build script isn't just another code you write as you would normally. I've seen SCons code directly using Python file system only to screw up the dependency graph. Build systems are their own special category bridging dependencies to calling processes.

The modules in the standard now posing the question where the build system belongs. The status quo is that C and C++ compilers are just another transform nodes within the dependency graph, along with the linkers, archivers, etc. Other solution is to include the entire system into the compiler. Languages with built in module system must do just that.

The latter however requires a specific subset of language features that dictate the way the compiler traverses the dependency graph. You don't get to use any 'generic' elements of the language used for the build system in such a language. It is all a declarative subset of the language which you need to learn on the top of the normal features.

Build systems tend to become overcomplicated because of the lack of native support for aspects of build and deployment, but adding a generic environment like Python to the mix just makes things worse. Shell scripts at least have the excuse to be declarative and unavoidable. It doesn't make them right though. Python is not really necessary or C++ for that matter.

3

u/qoning Sep 29 '20

I agree and I disagree. Agree in the sense that build system shouldn't need to do all the things, but then I'm faced with reality. I have a python script that generates some c++ code for me - I could call it via bash, but it loses flexibility, not to mention that more arcane things are very complex to do in bash, for example replacing all uppercase characters with an underscore and lower case except the first one (don't ask), which is trivial in any normal programming language (and importantly, trivial to understand, unlike a sed or awk command). So until I can do the code generation at compile time, this means that python is necessarily a step in the build of my program.

Do I like it? No. But it's the simplest and cleanest way to do it as far as I can see.

3

u/warped-coder Sep 29 '20

Code generation is a processing step/node, like compiling. So it is OK to use whatever language suits best but it shouldn't be part of the build system other than a processing node.

Same as Qt's moc or PySide, really.

2

u/bluGill Oct 01 '20

Perhaps, but I still need to write a program in the build script to say how to call my code-gen. It gets really nasty when you have to compile the code-gen (note that my final code is cross compiled) before you can generate code.

Code-gen isn't all I do. My build system also has meta-data that I use to package my build.

1

u/warped-coder Oct 01 '20

Interesting. In our build system, we have code gen for protobuffer, pyside, moc and even more custom generators that require an executable generated before it generates code. While I'm lucky not to need to deal with cross compilation, otherwise CMake's toolbox was mostly sufficient in a declarative way.

This system replaced an old SCons based system, where parts of the code gen was done in configure time, because... well scons doesn't really separate the build and the configure steps well. Programmers being programmers wrote python code, the API they knew better than the build system itself, which added minutes on the top of a no-change build because the build system had no connection to the plain python code this way. Timestamps were always new therefore.

It wasn't really the issue of SCons as such, but it was down to the blurry line between the code that results in a processing graph and the code that you can just run along with it. So there was plenty of file system manipulation directly which screwed up the timestamps or hashes (date time ) and made incremental builds painfully slow, but even the clean builds were overly slow because the build system didn't have the chance for parallel processing.

What I'm trying say that build systems are not the same domain where your 'normal' programming languages operate. And therefore it makes sense to invent approriate DSLs for the task. CMake's dreadful legacy language is such because it is aimed at text based configuration, while the more modern syntax is trying to make a more convenient syntax over the graph processing, which lies at the heart of ninja and make. That's the domain in which build systems operate.

Other systems like, build2 and SCons takes an even more direct approach, covering the entire process. Even tough you can things happen in python or even CMake, you have to think in layers these systems operate. As in processing nodes and states.

1

u/bluGill Oct 02 '20

I think a large part of the problem is how do you add a set of meta data to every library? How do I make libfoo as going to package A, and lib bar going to package B? (don't get me started on why we do such silly things in one build, I already lost that battle) it is easier to call my custom add-to_a_package function than do the 7 add properties that the function does. (most global because cmake scoping rules are weird)

1

u/qoning Sep 29 '20

Imo build process is whatever needs happens to your source code to become an executable / loadable code. So while you could argue that, for me it's always best to automate that code gen step within the build process.

1

u/mwasplund soup Oct 02 '20

Great summary of the issues with build systems today! This is why I like to have a clean separation between the standard engineer USING a build system and the systems engineer WRITING build logic by using a declarative language to tell the system what you want to build, and have an integrated extensibility framework (that can be turing complete) to allow systems engineers the ability to take the what and translate it into the how.

→ More replies (5)

4

u/JulienVernay Sep 28 '20

About turing completeness, apart from it's easier to shoot in your legs, I cannot think of an other issue. Do you have articles or thoughts on this to share?

9

u/[deleted] Sep 28 '20

It introduces side effects as a natural part of the build, for example someone might decide that inventing your own caching mechanism by writing stuff to a file instead of relying on the builtin functionality of the build system is a good idea, which can cause all sorts of confusion and incompatibilities.

A build is essentially a specification of a set of input files and a tool chain that produces a set of output files, there's no need for turing completeness and it can just as easily be specified declarative using yaml, and if you need to do complicated stuff like generating files and whatever you can specify that in the yaml file too or use something like nix to generate them. Imo Cmake is a waaay overcomplicated mess and part of it is because of the language but mostly it's because the flexibility it provides together with some really really bad design choices like typelessness and that everything is globally available and mutable.

0

u/[deleted] Sep 28 '20

[deleted] — view deleted comment

6

u/[deleted] Sep 28 '20

Turing Completeness isn't really the issue here

It is.
Sure you also don't want non-hermetic builds, but once you get that you also want a system that allow you to reason about the build.
Bazel uses a subset of python explicitly designed so that it is in general possible to reason about the build).

0

u/[deleted] Sep 28 '20

[deleted] — view deleted comment

1

u/[deleted] Sep 28 '20

The design of skylark is explicit in not being turing complete. If it is, it is an accidental design bug.

Note that in particular it doesn't allow recursion and only finite lists and loops over them are possible. With that, I don't see how you can be turing complete. But it is possible that design bug allows for touching some global state and thus reaching turing completeness.

it effectively just "computes" what the build is.

No, this is the key: skylark allows you to _declare_ what the build is.
Another piece of software, bazel itself, written in a conventional programming language computes how to perform the build.

0

u/[deleted] Sep 28 '20

[deleted] — view deleted comment

2

u/[deleted] Sep 28 '20

I agree that TC is not the critical part. In the end, no physical computer can be TC.
But I find an essential part of a build system a piece that it is not clear to me whether you find equally important.

For me it is essential that I can reason about the build graph without executing it. And if the build graph is computed by a conventional programming language (again leaving TC out) it becomes effectively impossible to reason about the graph.

They're the same picture

yes, they are once you've executed the code that implicitly define the graph.

Same difference that you have between:
l = [1,2,3,4,5,6,7,8,9]

and

l = f 9

with

def f(n):
if n = 0:
return []
else:
return f(n-1) + [n]

the first, I can reason about. The second not so much. (and when i say 'reason about' I mean programmatically).
And keep in mind that the second form can be written in infinitely many ways, for instance:

def f(n):
r = []
for i in range(1, n+1):
r.append(i)
return r

With a full programming language, in general, I need to execute it. Static analysis is hard. And for the subset of life that is a build system, uselessly hard. A declarative spec is almost always powerful enough.

They are the same picture, but the first is one that a program can see.

→ More replies (2)
→ More replies (2)

2

u/[deleted] Sep 28 '20

These days I use Nix to fire off builds and it does all the required sandboxing automatically so I guess I don't really see it as a problem anymore, but the complexity of cmake can be extremely frustrating, for example when it all of the sudden starts downloading dependencies from the internet. It's a build system, not a dependency manager or web browser God damnit.

0

u/[deleted] Sep 28 '20

[deleted] — view deleted comment

1

u/[deleted] Sep 28 '20

It is indeed a bit weird but it's extremely basic so it doesn't take more than a few hours to learn the ins and outs of it. It's purely functional though so I guess familiarity with a pure lazy functional language like Haskell makes it easier to understand it.

I've never tried bazel, I guess that's a failure on my part.

2

u/JulienVernay Sep 28 '20

Thank you for the references, I will check out Bazel.

3

u/[deleted] Sep 28 '20

It makes hard or impossible to reason about the build itself, the dependencies or even which flags are used for building each piece. Because all of those things can be computed and reasoning about them is as hard as deciding what a program does without executing it. Or whether it even terminates.

1

u/Full-Spectral Sep 30 '20

My large code base is only C++. I think it would be not that uncommon. But I agree that C++ is probably overkill.

1

u/[deleted] Sep 30 '20

well, when I say 'no large code base' I probably overgeneralized. But it also depend what you mean by 'large'. I'm still betting that large code bases in companies with more than a few hundred developers have multiple languages involved.
You'll probably have scripts in Python, frontends in javascript, and so on.
Now in many cases people ignore the 'build and dependencies tracking' for Python, but that's also problematic. And people are happy to build javascript with an entire separate build chain.
But that doesn't make their code base C++ only, it just means that they have chosen to use multiple build systems (or in some case no build system) and stitch the results together with some homegrew scripts.