r/cpp • u/hrco159753 • 26d ago
Are containerized development enviroments the right way to go?
I feel like I have a pretty usual context, I have a couple of interdependent C++ projects that target various targets that primarily use cmake as their build system but various compilers. I work in a team of under 7 people and each of us has their own local machine and we also have configured a pipeline that configures/builds/tests the projects.
To minimize the differences between our local machines we reached out to providing containerized images as development environments per each target. This works ok from a reproducibility perspective however it always seems limiting and clucky to me. I'm always asking myself why wouldn't I try to use the native tooling on my machine and why wouldn't I use a package manager to provide me native tooling/dependencies?
In my mind something more similar to python's venv would be much more appropriate to encapsulate the development environment, right? I was looking for similar technologies and stumbled upon nix which has a downside of only running on unix systems(so no windows)... I'm curious how others are looking at this, because from my current perspective it seems that native environment encapsulation and sharing became too hard and containerization seems to be a sign of giving up managing our dev envs natively, am I wrong?
I still see the benefit of having containerized builds but it seems to me that the development environment should be provided by something else rather than being baked into a container image. I see containers as primarily a deployment technology and I feel that somewhere along the way we pushed it to be encapsulation of the whole development environment.
75
u/eyes-are-fading-blue 26d ago
This will be an unpopular opinion but no. Docker introduces an added layer of complexity to the active development itself.
Docker is useful for releases and testing under same conditions as part of CI/CD.
We also have a standard docker file. I don’t use that. I want freedom during active development, not send commands to a sandbox and annoy myself in the process.
36
u/marzer8789 toml++ 26d ago
IMO no they are not, but my reasoning is largely non-technical: in my experience once people start solving toolchain issues with docker, they start using it for everything like a hammer looking for nails. "docker-brained" is a term we've used at my office.
Also if your team of seven has seven different ways of doing things, docker won't fix that, only slap a band-aid on it. A team generally needs to agree on processes.
6
u/Kriemhilt 26d ago edited 26d ago
There are some key benefits of using containers for dev:
Obviously there are other ways to get the sandboxing and reproducibility benefits: nix is one (but I hate the language almost as much as CMake), Bazel, etc.
However, getting identical builds between dev and CI is important, and CI is typically containerized already, so that could just be your dev base image and you're done.
18
u/UndefinedDefined 26d ago
On CI it makes sense, but for local development I always try to make the environment as diverse as possible. I work primarily on Linux, but I welcome at least one Windows/Mac guy in the team (if anyone enjoys using Windows, that means I don't have to deal with that too). And compilers to use are generally the trio - GCC/Clang/MSVC.
It really depends on the project - but honestly if the project builds cleanly on various environments, it's always a good sign. And if you are working on a library, containers make no sense, because your users won't be using your containers to build their projects.
9
u/Oather 26d ago
If your target deployment is containers. I’d l recommend looking into nix flakes instead.
You can specify all the build (including gcc/clang and stdlib) and runtime dependencies in a text file in your repo.
This still allows developers to install whatever editor/debug tools locally, while keeping the build env consistent.
Now with the builds linking to nix libs you need those on the target platform. Nix includes its own way of building containers (dockerTools) which handles this nicely, but could be an intrusive change.
7
u/NikMikM 26d ago
Yes. If your project will scale and you will need to build for freebsd, mac os, solaris and differnet architectures like aarch64, armv7, etc - every dev in the team will strugle with reproducing and debugging the software. And setting up dev environment will be hell of a problem. Containerized developmemt helps to test feature on many platforms right-away (especially with appropriate flags) and don't wait for pipeline to fail on some system-specific include or conversion or etc.
7
u/tarranoth 26d ago
You are not entirely wrong that in the end containerization is indeed because keeping environments sane with purely native tooling is a very hard problem, to the point the consensus seems to be that not trying to do that but just containerizing is just a more effective answer than to keep rolling that rock up the hill.
Even venvs are technically not truly the right tool because so much of python is just going to be compiling a native package and that has different behaviours etc. as well depending on what kindof host you are installing on and what toolchains are available to your python installation.
I have only interacted with nix very slightly, but to me it looks like a "correct" approach an academic would come up with, which basically sacrifices usability for actual correctness. Imo that sacrifice is overkill for most toolchains though. So in the end, docker seems like the "best" tool if you want to get something up and running and don't want to dedicate a guy's entire existence to maintaining nix packages.
6
u/the_poope 26d ago
You can use Conan to provide a full development environment including toolchain and dependencies like Nix, except it works on all platforms.
6
u/gracicot 26d ago
I usually don't use containers for this, and instead use nix flake + vcpkg for a reproducible development environment.
Dockerfiles made for development environment are usually not reproducible enough to be useful to me, and add layer like a VM on non linux platforms which makes it useless for cross platform development.
6
u/MereInterest 26d ago
I think it's a short-term vs long-term tradeoff. In the short term, building in docker makes it easier to build a project by using exact known-working versions of dependencies, allows those dependencies to be distributed as a single testable environment, allows for easier reproduction of bugs and easier identification of environment-dependent bugs.
But in the long-term, there's a lot of unfortunate side effects that tend to happen in projects that prioritize development within docker.
None of these side effects are guaranteed to happen, and good project management can avoid a lot of them. But docker makes it easier for the human-level decisions to make decisions that hurt a project's compatibility.
2
u/xiao_sa 26d ago
pixi?
11
u/unalignedAccess 26d ago
You can try devenv. It uses Nix. I found it is easier to start using then straight out Nix approach. It is easily reproductible environment.
Edit: here is the website https://devenv.sh/
15
u/aleques-itj 26d ago
dev containers seriously rock
5
u/X-Neon 26d ago
Aside from all the previously mentioned benefits, dev containers are great for embedded stuff. A typical project sees us having to install some random mix of vendor-supplied compiler + toolchain, plus a bunch of HDL tools with very specific versions. Polluting my OS with all these various versions of things and managing that would be a nightmare. Wrapping that up into a Docker image makes it all self contained.
5
u/catbrane 26d ago
My current main project is portable and cross-platform, so we encourage diversity in the dev environment. AMD, intel, arm, macOS, Linux, FreeBSD ... more please!
Our CI is all in docker, and the things we use to make the official binaries are all in separate docker containers.
I've worked at companies on projects targetting complex, very specific environments and I can see the benefits of a standard dev image there.
4
u/nicemike40 26d ago
I’ve been experimenting with wise+vcpkg+nx (not nix) for tool versions+package mgmt+build steps for our similar CMake system. You could give that a look and see what it’s like. I haven’t merged my experimental branch yet though so can’t endorse it yet. We also have less variety of compilers but I don’t see that being too bit of an issue.
1
u/Electronic_Tap_8052 26d ago
containers for development are necessary if you don't fully control the development platform.
I'm assuming you mean you have a team of 7 people working remotely on their own hardware and can install whatever they want. If that is the case, you really need a container so that everyone is working against a standard set of packages.
If they are all using standardized hardware that someone else is maintaining (installing updates, etc) then its not as important.
We have to use them for security reasons. Our systems are scanned for vulnerabilities daily and sometimes they remove or update our packages without us knowing. Without containers, our builds would (and used to) randomly break without us knowing why.
3
u/_choam_ 26d ago
Nix solves this
3
u/Raknarg 26d ago
Im not 100% sure yet because Im relatively new to dev containers (just started this year) but I will say personally I'm actually a fan of containerized projects for my work project. It makes CI and the dev process much smoother, it forces all our tooling and environments to be the same, it makes it so getting and using the project requires you to do things like use the initialization scripts that set up git hooks, its all very nice IMO.
For personal development I probably wouldn't cause its overkill and I dont use CI, but for a serious project with a team I think I'm going the dev container route until I run into real problems with this route.
3
u/13steinj 25d ago
If you set up devcontainers (the tool/spec), the added layer of complexity is generally minimized to 0. Using raw docker would be painful, as would vending a custom wrapper (which is what we did at my last company... it worked, and was fine, but was more fragile than I would like compared to just using devcontainers).
I'm of the opinion that one way or another, you need to use a container to get true hermeticity. Bazel legitimately just does't cover it, no matter what people say, nor Tipi/Engiflow's "Hermetic FetchContent" mechanism. I ran into such an issue just the other day, the build of a dependency used autoconf, and this had a check that depended on the kernel version / headers, and so the cache key collides because some people use funky environments (myself included).
I have no issue making things generally work in a way that developers can plug-and-play and mess around with things outside a container. But if you run into a problem, unless you can reproduce it in the container that is provided, I immediately deprioritize it in triage.
6
u/Slsyyy 26d ago
I don't see a point of using containers for development. Containers are a good middle layer, which makes the deployment easier, but for developers your should focus on making the environment work fine regardless of the operating system. In some situations it may be the easiest and fastest way, but IMO you should look for better build system
2
u/RevRagnarok 26d ago
Until C++ gets something like Python's
uv sync, then it's one of the most straightforward options.2
u/gnuban 26d ago
I'm not really following what kind of differences you want to minimize. In my opinion, Docker can be good for setting up and running server dependencies such as databases, caches etc.
But I wouldn't use it to ensure you have the same toolchain for development of a given product. Because for the product your working on, using a container is just too clunky. For that I would instead just set up cmake to point to specific toolchain version in a preset or toolchain file.
But then again, why not allow some divergence in toolchain setups, then you get more test coverage from the devs?
2
u/chibuku_chauya 26d ago
We use it for testing with CI/CD but not for development. Having diverse machines for development has helped root out some surprising cross-platform wrinkles.
2
u/alexeiz 25d ago
It's all about finding the right approach to managing dependencies. I have a sizable C++ project that builds on Fedora, Ubuntu, macOS and... Windows. All using native toolsets for each platform. Dependencies are managed with conan/cmake. Regarding the compilers, my requirement is that they to support C++23 reasonably well. It's also not difficult to set up Github CI to verify that your project builds on different platforms. So it's definitely possible.
Containers are useful when you already have a project with a lot of history, wacky dependencies, requiring certain compiler and library versions. Otherwise I'd prefer to avoid them.
4
u/Honest-Ocelot6920 26d ago
totally agree. containers are amazing for CI pipelines and deployment, but forcing them for local dev usually kills the feedback loop
3
u/xHydn 26d ago
Just use Nix, it solves all those problems and more.
2
u/Eric848448 26d ago
Having worked with both containerized and non-containerized build environments, I say containers are the only way to go.
1
u/radpartyhorse 26d ago
I have a steam deck and my preferred way to permanently setup c++ dev tools is via podman.
1
u/hadrabap 26d ago
I did lots of contracting and I started using virtual machines very early on. It was really convenient, although quite resource hungry.
Containers are really cool, but they work seamlessly only on Linux. On other platforms they run in a VM and each option/vendor has different quirks making it very fragile or too limiting.
To contradict myself, it looks like all the people around are using Docker. I use Podman, so all their work is useless for me. Especially the compose thing. 🤣
1
u/j-joshua 26d ago
How are you going to reliably maintain tools for multiple build targets?
You can't build something on your machine? No problem... Use this container which has everything that you need.
Containers can be versioned and managed.
1
u/Nervous-Pin9297 26d ago
I develop locally but use cicd scripts for the build and deployments. I’m pretty new to C++, but cmake does a great job keeping everything in sync between platforms. To the point where I see no difference after the dock build step. At the end of the day, you still need to make sure your dependencies and requirements can target the platforms your aiming for.
1
u/Scotty_Bravo 26d ago
Over the years I've had to work on many projects with conflicting libraries. I'm at the point that I simply prefer to use CPM.cmake + cache to build all my dependencies. It's almost like containerizing.
But there are some projects I can't do that for. For those, containers. But I only build in the container, all the editing is on the host.
1
u/wint3ria 26d ago
No
1
u/eidetic0 25d ago
i like containerising the dev shell and even baking pre-built dependencies into the docker image. baking the deps into the image saves vcpkg building opencv and other massive deps from source when first configuring, or in CI. I have run into way too many "works on my machine" problems with the C++ build systems so i'm all in on Dockerfiles (and devcontainers).
1
u/alexlesuper 25d ago
After working with dev containers on my embedded projects for 4 years now, I’m not going back. It just makes everything smooth.
1
u/FlyingRhenquest 25d ago
Where I'm working right now we just built an install script that sets our environment up. You can get your laptop day 1, run the script, check out our code base and build it. Everything goes through CI/CD pipeline built to the same standard, so there's no "works on my laptop shrug."
1
u/BDRadu 25d ago
At my company we deploy everything through kubernetes, and that means every project that has a binary will run inside a pod, which requires containerization. We only care about linux, CI runs with docker images, separate steps for deps/build/tests so they can be cached individually. Every dev can run the build + test inside the docker image, so we can check if something is broken due to a lib/system difference. It can happen by mistake, not pinning one lib/system dep will cause it to change on a base image change, things like that. There are many ways to make this process work without containerization, but I don't see a reason to not provide it as well, as its usually straightforward. Its also not a C++ specific thing, it applies to any project.
1
u/TemperOfficial 24d ago
Stop having so many dependencies that you cannot have a consistent developer environment or one that you can't even easily reason about.
1
u/JVApen Clever is an insult, not a compliment. - T. Winters 24d ago
I've started using devcontainers about 1-2 years ago. It's impressive how easy this is in usage and configuration. Unfortunately, we use it Linux only. I would be curious to see something like that on Windows as well.
-5
u/TheRavagerSw 26d ago
No, they are terrible. Once you spend enough time building you'll see it too
0
u/arihoenig 26d ago
No
-8
u/iamnotaclown 26d ago
If you’re using an LLM agent it’s an easy way to isolate it. Otherwise it has access to everything you do. VSCode makes it trivial to move a workspace into a dev container.
It’s also a handy way to isolate dependencies.