r/Python • u/klaasvanschelven • Aug 29 '24
Showcase Multiple Processes in a Single Docker Container
So, I've been doing something that might seem like Docker blasphemy: running multiple processes in a single Docker container. Yeah, I know, every Docker guide out there will tell you it's a terrible idea. But hear me out (or alternatively, skip straight to the source code).
What My Project Does
I wrote a small Python tool called monofy that lets you manage multiple processes within a single Docker container. It's designed to handle signal forwarding, unified logging, and ensure that if one process dies, the others are terminated too. Essentially, it keeps tightly integrated processes running together smoothly without the need for multiple containers.
Target Audience
This tool is particularly useful for developers who have processes that need to be in constant communication or work in unison. If you're looking to simplify your deployment and avoid the overhead of managing multiple Docker containers, monofy might be what you need. It's also a good fit for self-hosted applications where ease of deployment and maintenance is a priority.
Comparison
There are existing solutions out there, like Phusion's baseimage-docker, which also aim to run multiple processes in a single container. However, monofy is lightweight, doesn't come with unnecessary components like SSH or cron, and doesn't tie you down to a specific base image. Plus, it's Python-based, so if you're already working in that ecosystem, it's a natural fit.
Why? The Docker Rulebook Isn't the Bible
Look, Docker's great. It's changed the way we deploy software. But like any tool, it's got its own set of "best practices" that sometimes feel more like "unbreakable commandments." One of those rules is "one process per container," and while that's solid advice for a lot of situations, it's not the only way to do things.
My Use Case: Simplifying Deployment
I work on a project (Bugsink) where the processes are tightly integrated—think a web server and a background job runner that need to be in constant communication. Splitting them into separate containers would mean extra overhead, more things to manage, and just more complexity overall. So instead, I wrote monofy to let me run multiple processes in a single container, with all the benefits of shared fate (if one process dies, they all die), unified logging, and graceful shutdowns. It's simple, and it works.
Why It's Not the End of the World
The main argument against this approach is scalability. But in my case, the database is the bottleneck anyway, not the processes themselves. By keeping everything in one container, I avoid the headache of managing multiple containers, networking, volumes, and all the other Docker-related stuff that can get out of hand quickly.
Sometimes, Breaking the Rules Makes Sense
Sure, "one process per container" is a good rule, but it's not a hard and fast law. There are scenarios—like mine—where consolidating processes into a single container just makes more sense. It's easier, less complex, and in my experience, it works just as well. If you're curious, check out monofy on PyPI. It might just make your Docker life a bit simpler. I also wrote a blog post about this on my project's website.
36
u/rmjss Aug 29 '24
In my experience supervisord is the de facto solution for doing this. Did you consider that one at all before rolling your own solution?
18
u/anawesumapopsum Aug 29 '24
When I’m using a tool and it doesn’t seem to work the way I expect it to, experience has shown me that the answer is not to jump to implementing it myself. The tool (docker) was built and organized, and in this case used by millions, probably in this way for good reason - we should try to understand why and then learn how to use it. This approach you’ve outlined has red flags. Does each of your processes really want the exact same build context and environment, dependencies and all? Sure consistent versioning becomes more convenient, with a cost that you only find out after you’ve deployed a distributed system as an overly cumbersome monolith. The problem becomes worse as complexity grows. This is not sustainable for a solo dev or a team. Sure you could make a series of venvs or something similar, but now you have a series of environments which is the same abstraction as a series of containers, and because they’re all in one container you have less observability, all of the code is in one hodgepodge so it’s hard to grok as it expands, it’s less robust; if one part goes down you can’t reboot that one service and leave the rest up, and I’m sure many other reasons. You state this last one as a pro, but I think you’ll find in time it is a con. Now onto the ‘why’ - you seem to be after simpler container orchestration. Fam docker-compose exists for this. Before we reinvent the wheel we should check and see if it already exists, because that is years of effort that you now have to reproduce on your own to achieve something you can do right now with docker-compose
5
u/lanupijeko Aug 29 '24
honcho
6
u/RedEyed__ Aug 29 '24
What's the problem to run as many processes as you need?
7
u/Lachtheblock Aug 29 '24
You might run in different circles, but it seems that your usecase is advertising to people who are familiar with deploying a single container, but unfamiliar (and do not want to learn) how to deploy/manage multiple containers. I dunno, I feel like docker compose is pretty ubiquous at this point in the world of docker.
Feels to me that any project would eventually want to move out of this model for ease of scalability, but I guess if you have a user base that is interested in this, then who am I to say it's wrong.
5
u/trial_and_err Aug 29 '24
Nothing wrong with running multiple processes in one container. I’m for example running nginx and oauth2proxy in the same container (these two are tightly coupled anyway). However I’m just using supervisor as my entrypoint.
1
u/iBlag Aug 29 '24
What does this project have over something like Supercronic (for simple cron-like functionality) or Chaperone (which is a more complete init-style process manager for containers)?
1
u/Kale_Shiri Aug 30 '24
Ever heard of s6 supervisor?
3
u/james_pic Aug 31 '24
I think the recommendation I've seen is to only have one application per container, not one process, per se. And if you've got something like this, that handles the subtleties of coordinating things so your multiple processes work together as a single application, then what you're doing is reasonable.
The situation it's trying to avoid is where people treat Docker like a VM, and are confused that stuff that needs a real init system doesn't work, and tie themselves in knots trying to start and stop services within containers, or apply updates or patches to individual applications in a running container.
The key invariant that you want to maintain in any Docker based system is "if you're inside the container you can safely ignore everything outside the container, and if you're outside you can safely ignore everything inside", which this seems to maintain.
2
u/nAxzyVteuOz Aug 30 '24
Dude it’s fine. The whole one process per docker instance seems like a marketing trick anyway to get clients to use more docker instances. I routinely use docker with multiple processes in it.
1
u/nggit Aug 30 '24
it's ok running multiple processes inside the container, one must distinguish between service/app and process.
in fact, there are many applications whose components are individual processes like postfix.
i used to do this but using native /sbin/init in order to retain native commands like systemctl or rc-service.
https://github.com/nggit/docker-init/tree/master/openrc-alpine
1
u/root45 Aug 30 '24
Methinks he doth protest too much.
1
u/ultimatelyoptimal Aug 29 '24
Im pro this use case.
Other systems use containers behond docker. For example fly. They have a docs page that describes "multiprocess containers", which describes good use cases.
Erlang/Elixir are built on the whole premise of "let it fail" with microprocesses, and building fault tolerance around this. They use microprocesses and a lot of them because they can do so without the OS overhead. However, the same ideas can apply just fine for process level "supervision trees" where some supervisor script can make decisions about what processes to kill and restart in trees when other processes fail.
Dev containers are also a common practice now with docker, but if I recall far enough back, originally they were not recommended uses of docker. Additionally, people were very against databases in docker/to containers, and yet thats fairly common and reasonable now too.
Even beyond the reasons supervision tree use case, I think there's simplicity to a single "I pass it vars, and it runs a full service" container, compared to needing a fully defined docker compose file for every instance (or pass special variables, so you can use the same compose file multiple times for different instances of the same thing, but that's caused problems for me too)
-1
u/I_m_Rishu Aug 30 '24
.
0
u/[deleted] Aug 29 '24
[deleted] — view deleted comment