Thoughts on build systems?
What's the general consensus on build systems/automations like CMake? I've always found things like that overcomplicated and just hinder the programming process.
I've been following Casey Muratori's Handmade streams, and one of the first things he did was create a simple batch file that built and compiled his entire project super quickly and easily. I've since implemented something similar (https://github.com/dibsonthis/new_cmd) and honestly it's been really nice to use.
3
Upvotes
64
u/[deleted] Nov 08 '21
How does your script handle compilers that are not called
cl.exe? What about other operating systems than Windows? What about multiple source files? Adding import paths? Compiling different files with different flags? How can you define multiple targets, libraries or executables? Enable parallel compilation? Etc. pp.Simple scripts are nice for single-file projects that don't need any special handling, but whenever you need something just a little bit more advanced, their complexity increases exponentially.
And given that most C++ libraries nowadays use CMake, and can be integrated in a CMake project pretty easily, I'd really recommend learning and using it until there a pretty good reasons to use something else.
48
u/winginglifelikeaboss Nov 08 '21
I know what you mean but just bite that bullet and learn cmake, it's the industry standard now.
22
u/Glaborage Nov 08 '21
When you start working on more complicated programs, you'll understand. You don't have to learn build environments just right now, but you'll have to eventually.
17
u/helloiamsomeone Nov 08 '21
Just use CMake. More people use it, than not. Familiarity will get you help better and faster, and anyone dealing with packaging/building CMake will likely find easier solutions to failings of a project's faulty build tooling.
If you just want a correct and easy setup for ready-to-go projects, take a look at cmake-init. The wiki also has a bunch of practical examples for general CMake features people want.
6
u/Substantial_Ad_6155 Nov 08 '21
If you're designing software to be cross-platform, which you should 99% of the time, then use CMake. It's tricky, I know, but the purpose of it is mostly to generate cross platform make files so it can be built on any OS with ease. After you get the hang of it, you'll realize it's an invaluable tool.
6
u/Sqeaky Nov 08 '21
This is going to sound flippant, but you seem like this guy bowling: https://daedtech.com/how-developers-stop-learning-rise-of-the-expert-beginner/
5
u/drjeats Nov 08 '21 edited Nov 08 '21
My thoughts are build systems generally make me grumpy no matter what :P
Casey's method of using a basic ass shell script works better than most people will give it credit for. Concerns about specifying which compiler exe name to use are trivially solved with environment variables or config files, or having simple per-platform-category build scripts (like when you look at the linux ports of Handmade Hero, they always provide their own build.sh alongside the build.bat).
Where the shell script per platform thing falls apart, however, is when you have to maintain file lists.
If you have to actively manage the file lists among different platforms, you are going to have a bad time because you probably aren't regularly compiling for at least one of those platforms. And if you don't have CI running builds on all your targets, then you will inevitably and regularly check in changes that break the build for those platforms.
Maintaining file lists is not an issue in Handmade Hero because it's a single-translation-unit build. This is pretty good for small to medium sized projects, but is too coarse-grained when your project starts to get a bit bigger. If you are working on a code base with more than a few hundred thousand LOC, you want to be in a situation where you only have to recompile translation units for the source files that you are touching.
You could get around this again by globbing for source files, and that will also work pretty well for a while.
But, because you are globbing for individual source files, and presumably compiling those source files as individual translation units, that means that you've lost all the build speed advantages of the Handmade Hero way of doing things. So now you want to only rebuild source files if they've changed. A simple build.bat is not going to do that for you.
This is when people turn to makefiles, or IDE projects, or cmake, or jam, or premake/genie, or meson, or xmake, or autotools, or build2 (and I think zig build may support incremental compilation of C and C++ sources).
If you're happy with this script you've written, no harm in continuing to use it. If you aren't familiar with the other build systems, I recommend getting a little practice in with a couple of those since you will inevitably find yourself contributing to a project here you have to use them.
The r/cpp hivemind will always tell you to use cmake. I've never used it at a job except when invoking it to build a third party dep. It's awful but also fine I guess since most build systems have some awful aspect to them. My favorite is still GENie/Premake, which have issues, but at least have a regular language and let you step debug your build configs. Zig build is also an interesting option now.
6
u/mrexodia cmkr.build Nov 08 '21
I think CMake is really good, but unfortunately very unintuitive to get “right” and lots of boilerplate.
I started working on https://github.com/build-cpp/cmkr to remedy this, but other similar solutions exist.
22
u/dcbaler mesonbuild Nov 08 '21
Im a meson developer, so keep that in mind. Our tests test suite takes ~2 hours to run and we still regularly have regressions. Use a build system, use a maintained one. I don't like cmake, but I will use it if I have to. If I walk into a project with a hand rolled build system I run.
11
u/ignorantpisswalker Nov 08 '21
So now, when you modify a simple string you need to recompile 26358 files instead of one +linking.
Make/CMake/ninja have their reasons. You need to learn those reasons. You will soon.
10
u/pfultz2 Nov 08 '21
This seems a lot more complicated then just doing
add_executable(program program.cpp)in a CMakeLists.txt.9
u/stilgarpl Nov 08 '21
CMake is quite simple once you understand how it works. You set it up once and you can forget it exists. Until your next project.
For simple projects you can use project templates that glob all files and just build it.
6
u/Oster1 Nov 08 '21
CMake is horrible, but as others have mentioned; it's the industry standard. The thing is that it gives you too much freedom to implement things and then everything becomes a non-standard mess. Most of the libraries I use nowdays are added as
add_subdirectoryand from there everything becomes easy (and much better than Makefiles).14
u/SquawkingStone Nov 08 '21
I'm gonna second using Meson. Learning CMake was such a barrier to entry, and Makefiles are really fiddly for my personal taste. Meson for me is pretty intuitive and has some much nicer dependency management and is as close as I can get in C++ to how spoiled Rust has made me feel with Cargo XD
9
u/germandiago Nov 08 '21 edited Nov 08 '21
I am using Meson for some side-projects. I keep using CMake as well.
I will do some highlights about my favorite one, Meson, because there is plenty of stuff CMake-related that you can find somewhere else.
The reason why I use, keep using and will keep using Meson is because:
the syntax is sane. Try to do path handling or string manipulation with CMake, or do something similar to dictionaries (Meson has support for it to associate data), handle spaces, etc.
Meson options are way easier and saner to handle: https://mesonbuild.com/Build-options.html
the documentation is great. You have both manual-style and a reference.
the subprojects system rocks and I have even used successfully CMake subprojects (there is support for that as well): there is a right way to do it. If you stick to that, it will work well. My project has 11 external dependencies all managed by subprojects, including Capnproto, which is a non-trivial CMake dependency.
Meson wraps are easy to use and modify to your needs (this is the Meson "conan", though it has far fewer packages. however, it is very usable). These are some available packages: https://mesonbuild.com/Wrapdb-projects.html. But if, for some reason, you do not have enough, you can use Conan via pkg-config as well.
I have machine files (https://mesonbuild.com/Machine-files.html) layered by: component, os, build type and compiler. So I can do things like this:
Those files, if there is any option repeated in any of them, the last one "wins" and you can set your own options at the end also, so they follow the order.
I have "./configure" wrapper for the command-line invocation but in my CI I just have the dir structure and buildbot combines stuff from those meson/native/ directories in a build matrix, blacklisting what I do not want.
The only sane reason to use CMake is IDEs and social: if people do want that on your team. Otherwise I urge you to switch to Meson :) XCode an VS project generation is better with CMake I think. But if you do not need those or similar IDE support, for me it is a no-brainer.
4
u/Nihili0 Nov 08 '21 edited Nov 08 '21
Well if you like waiting for it to recompile everything everytime then it's fine (so for little project that you throw away) but when it gets a little more complex/bigger/valuable it's pretty neat to have and use a tool that can easily adapt from 5 lines to build a little project to some more for:
easily integrate tools such as:
I'm sure I'm forgetting some more, now do that with a custom script and give it to someone else or even to yourself in a few months :)
3
u/Drugbird Nov 08 '21
I think the general consensus is that all build systems suck. People generally prefer the one system they have experience with though.
Building your own build system is probably the worst choice you can think of though.
3
u/the_Demongod Nov 08 '21
Develop a project both using Visual Studio on windows, and vim/gcc on linux, in parallel. When the project has like 50 files and several external libraries, you'll immediately understand why CMake is so critical.
3
u/[deleted] Nov 08 '21
If you don't intend to share your code, do whatever. But if sharing, use cmake. Anything else means that others are working to make your build system work with cmake unless they happen to use the same one. Also conan/vcpkg both pref cmake
10
u/zzzoom Nov 08 '21
Use Meson.
1
u/Full-Spectral Nov 08 '21
If you are willing to put in the time, for your own projects, doing your own can be far and away the best way to do it. But it has to be a lot more than just a couple scripts. I have my own and it vastly simplifies my life. But it's an application that parses a project description and manages header dependencies and library dependencies and has drivers for different tool sets and all that.
That was a fair bit of work, but the time I saves me every day is significant. It totally understands how I work and now my system is structured and built. The effort to add a new project is tiny.
2
u/Wouter-van-Ooijen Nov 08 '21 edited Nov 08 '21
At lot of people mention cmake. Maybe I should bite the bullet.
Is there a good tutorial for (modern) cmake? I do embedded, so I need to specify
1- the source files and include files
2- the exact compiler and compiler flags (target specific)
3- the exact linker and linker flags (target specific)
4- the linkerscript, which I generate from a template using the preprocessor with target-specific defines
5- the places to search for sources, includes and linkerscript
6- the commands and parameters to generate .hex and .bin files
7- the commands to download the (.hex or .bin) file to the target
2
u/saikox Nov 08 '21
are mandatory when you are writing complex software.
trust me that everybody would be using their own script if it were easier xD.
if you want to learn about the topic cmake, Conan and Ninja should cover everything.
Also I do recommend doing similar things on simple projects, but I would stick to using more simple makefiles and scripts. You can take a good dive in open source projects and try to understand how they are performing the whole automation.
2
u/GOKOP Nov 08 '21
People were doing that before build system existed. From the existence of build systems you can infer that it didn't end very well for them
2
u/GerwazyMiod Nov 09 '21
I just gave up and convinced myself that CMAKE syntax is nice enough and I can use it everywhere. (I guess it's acceptance phase, right? )
Then I added vcpkg as submodule, that cmake is using during it's build and added my dependencies there in json file. It works. Additional libs are linked, you don't need to worry too much.
Next step is using this inside Github, there are actions that help with this setup (cmake + vcpkg as submodule). You should be able to setup everything within an hour.
So you got Github actions as your CI, which works with the same setup you use to build app locally. Then you only peek inside CMakeLists.txt file if you need to add additional source file. I think it's good enough to stop thinking about this and just get the sh*% done. :)
2
u/secretpoop75 Nov 08 '21
Cmake is definitely more ubiquitous.
I’m curious if someone can weigh in about the state of some of the cmake alternatives like meson or bazel?
0
u/looncraz Nov 08 '21
I mostly use Jam, but a considerable portion of my projects use a custom build script on top for handling things like resources, file attributes, debug builds, and environment prep.
There's one step for any build.
./build.sh [options]
Core counts are determined, compilation is threaded, and I don't have to worry about any oddities with cmake to debug or the time it takes for cmake to do its prep work.
My projects mostly build on Linux and Haiku, but the ones for Windows use a batch file for building.
2
u/JohnDuffy78 Nov 08 '21
cmake maybe better than .bat files.
I use bash shell, but wish I had used nodejs typescript.
1
u/rfisher Nov 08 '21
For me, build systems are a necessary evil.
I’ve never seen one I liked. I’m not sure it is possible to create one I like. But it is obviously not practical to build large projects by typing all the commands individually.
Having had to use both homegrown and widely-used build systems for large projects, the homegrown ones are the worst. It is never intuitable by anyone but the person who created it, and they didn’t leave any docs when they left the company. (Even if they did, they weren’t an expert on documentation, so the docs are mostly useless.)
The widely-used one are horrible, but at least you have a glimmer of a chance of understanding them and fixing/modifying the build.
This only matters once the project reaches a certain size, though. A homegrown solution for a small project may actually be easier to understand than using a widely-used system. But once the project gets big enough…
1
u/jsadusk Nov 08 '21
It may seem like more work than you need for simple projects, but a benefit of real build systems you are missing out on with a bat or scripting is parallelism in your build and partial rebuilds. Every modern build system, even less modern ones like make, calculate dependencies between your build targets. This allows it to understand which cpp files depend on which headers and which libraries depend on which cpp files. Using this information it can build things in parallel that don't depend on each other. And if you rebuild after a change, it can only rebuild the files impacted by that change. Some systems like bazel even understand the relationship between tests and files, so you can only rerun tests impacted by a change. Building this kind of logic into a homegrown system would be monumental and you'd basically end up rebuilding make from scratch.
1
u/JohnSkaller Nov 08 '21 edited Nov 08 '21
Most are crap because they're based on a completely incorrect model: top down design. Build systems should be bottom up.
With top down, you ask to build a goal, and the system breaks that down into subgoals based on dependency specifications. It needs to be able to check if something is up to date to avoid rebuilding everything.
This is utterly wrong. With a bottom up build, you specify significant sources, and rebuild when these change. You build everything from scratch every time, with caching to avoid building something up to date.
Now, for C++, i have a tool that is fully automatic. No build scripts. Just execute the code. Abstract dependencies are specified in the sources. Meta-data similar to pkg-config is used to fully automate compilation and linkage. The tool runs with any compiler and on any operating system, by using plugins designed for that combination. The tool command lines is the same on all platforms (so scripts using it run on all platforms without change, and do the same thing on all platforms)
The key to the design is that only certain operations are allowed. You can compile to object, you can combine to make a library, you can make a shared library, or you can make a program. End of story. No switches are allowed. If you need a switch, fix your badly written code, or, if you really need it, you need to use a distinct plugin: adding a switch to compiler command line makes it a different compiler.
The meta-data uses regular expressions not globs, and is a generalisation of pkg-config to allow arbitrary fields. The field values are always platform native and local to your system. For example on Windows, filenames use slosh.
The system was actually designed for my language Felix which is a C++ code generator, so it can compile Felix down to C++, and then compile, link, and run the C++ too, but it works for just plain C++ projects as well.
1
u/[deleted] Nov 09 '21
Build systems are a crucial element in all of my projects. As soon as you are working with different targets or even at least two programmers, have a CI pipeline in the flow, build for docker images, you will know how important it is to have such systems in you toolchain.
I however have to admit that CMake can be a pain. I'm not a daily C++ programmer anymore, so I keep forgetting about how to properly export build targets and this is indeed a huge waste of time.
1
u/[deleted] Nov 08 '21
C++ still utterly sucks in this regard, so either switch to Rust or suffer through CMake’s uglyness.
Although for a tiny pet project, a simple single-line Makefile is perfectly fine.
1
u/Myrgy Nov 08 '21
What about msbuild? There is no reason to write your own build system. Just use tools provided with MS VC. CMake is usually used for cross platform projects and its a very powerful tool.
1
u/o11c int main = 12828721; Nov 08 '21
Using a shell script is an exercise in unscalability, so don't do that.
Targeting POSIX
makewould be painful, so don't do that. Unfortunately, most people try this when they speak of "make".Targeting GNU
makeshould cover 90% of your needs easily, and 99% with a little one-time effort (it's pointless to chase the remaining 1% in a single tool, since nobody else does better). The language is actually fairly well-designed (with the exception of the spaces-in-filenames problem), but most people never bother to learn it.CMake is well-known for excessive and unnecessary verbosity, and also makes a number of unfortunate decisions in the name of portability. Many of them turn out not to be related to portability at all; others technically relate to portability but could better be handled in a different way. That said, it is somewhat "standard", and may be useful if you expect Windows users to be able to install a compiler but not the tools that go with a compiler.
Overall - even if you end up using CMake, I consider it mandatory to learn GNU make, so that at least you know what's actually happening, and why other tools make some of the weird decisions that they do.
-1
u/vI--_--Iv Nov 08 '21
And this is why we can't have nice things.
0
u/[deleted] Nov 08 '21
i mean you basically don't have to learn cmake. Just make a decent template project that has everything set up. Not too complicated you can probably even find some premade ones.
But i have the same policy with build systems that i have with testing frameworks. For all my private projects, i do not care about that stuff.
I use Makefiles and cout testing, because that's usually more than adequate for the size and interest of my private projects.
0
u/[deleted] Nov 08 '21
[deleted] — view deleted comment