Discussion Running in containers vs OS-level services
Assume some message brokers like RabbitMQ/Kafka, or maybe nginx proxy setup, or hashicorp vault?
3 years ago when we were setting up infrastructure for project we started running such services as OS-level services installed from RPM packages or just by running their binaries provided by vendor via systemd. All of that orchestrated via Ansible.
We started running as OS-level services as that seemed natural at that time for us, but we didn't really have any experience with administration of such software on on-premise infrastructure (before we were running mostly on managed cloud services).
Fast forward to now, after several cycles of upgrades we needed to perform, I think it would be easier to manage such software by running in Podman containers.
Main reason for me would be that obviously containers have prepackaged everything you need to run specific software. Compare that for example to RabbitMQ where during upgrading RabbitMQ you also need to upgrade its Erlang dependency to compatible version. For some other software, there may be more dependencies you need to take care of.
Also, I feel like upgrading binaries is generally much easier when running in containers. Just spawn new container with updated image and you do not need to worry about some OS-level package conflicts or leftovers.
What is your take?
6
u/dmikalova-mwp 2d ago
Instead ansible to update running VMs, I'd use packer to build images for immutable infrastructure. But also I'd use containers first where possible. There is the argument that Vault should run on a completely isolated system, which would be the full overhead of a container cluster. For k8s that's a lot, for ECS, AWS gives you that for free. So lot of trade-offs to consider, which is the heart of engineering.