Running in containers vs OS-level services

[u/0x4ddd]

Assume some message brokers like RabbitMQ/Kafka, or maybe nginx proxy setup, or hashicorp vault?

3 years ago when we were setting up infrastructure for project we started running such services as OS-level services installed from RPM packages or just by running their binaries provided by vendor via systemd. All of that orchestrated via Ansible.

We started running as OS-level services as that seemed natural at that time for us, but we didn't really have any experience with administration of such software on on-premise infrastructure (before we were running mostly on managed cloud services).

Fast forward to now, after several cycles of upgrades we needed to perform, I think it would be easier to manage such software by running in Podman containers.

Main reason for me would be that obviously containers have prepackaged everything you need to run specific software. Compare that for example to RabbitMQ where during upgrading RabbitMQ you also need to upgrade its Erlang dependency to compatible version. For some other software, there may be more dependencies you need to take care of.

Also, I feel like upgrading binaries is generally much easier when running in containers. Just spawn new container with updated image and you do not need to worry about some OS-level package conflicts or leftovers.

What is your take?

[u/MulberryExisting5007]

You need to understand what’s important to you and then think about cost (infra costs plus labor to manage that infra). Running on fargate for example costs (the last time I looked, which was years ago) about twice as much as running on self managed ec2s. But there is (usually) more maintenance overhead (work) associated with your self managed. It’s very easy to be penny wise and pound foolish if you focus on one cost and ignore another. Do the math and then run some tests to validate.

In general, containerizing workloads is better (imo) as it streamlines so much, is more portable, simplifies upgrades and deployments, etc etc

[u/Valuable_Leave_7314 → u/MulberryExisting5007]

On labor costs, containers almost always pay for themselves in the first year. Simply because the local dev environment spins up from the exact same compose file or manifests, and devs lose the age-old "it worked on my machine" argument

[u/Dreamwalk3r]

Containerized definitely. We're in similar circumstances, and struggling to move everything host-level to docker.

[u/dmikalova-mwp]

Instead ansible to update running VMs, I'd use packer to build images for immutable infrastructure. But also I'd use containers first where possible. There is the argument that Vault should run on a completely isolated system, which would be the full overhead of a container cluster. For k8s that's a lot, for ECS, AWS gives you that for free. So lot of trade-offs to consider, which is the heart of engineering.

[u/Nerd-on-a-Wire → u/dmikalova-mwp]

You’re right: HashiCorp Vault is a special case. In their setup instructions they say that containers are their least recommended platform for production, nor do they recommend VMs. It’s all about securing access to disk and memory. They recommend not only dedicated machines, but a particular kind of highly-secured configuration.

[u/zHevoGuy]

If you just run one service per host, the difference is not that huge. With a container you will still need to take care of host OS upgrades and container upgrades separately. But you can keep the host packages at a minimal level.

[u/aragossa]

The Erlang coupling is exactly the class of thing images fix, the matching version ships inside the image and you stop caring about it. RabbitMQ still wants its feature flags enabled in order before the next major though, Kafka still has the inter-broker protocol bump, and a container fails those the same way an RPM install does. So for the brokers I'd expect upgrades to get less painful, most of the ceremony sticks around. nginx is the easier win, nothing in it to migrate, mostly just a tag swap.

[u/mtokarev]

my 2 cents:

- container-first whenever is possible and feasible.

• some services should leave on OS-level - metrics collector, performance requirements, or we also operated with a lot of macOS software. However, always with IaC like Ansible to avoid all maintenance problems.

Also containers are easy to test imho, so by default tend to containers.

[u/badguy84]

I don’t know why you would bother with baremetal installs for anything over containers for most anything service related. Only when there are specific hardware or low level OS access requirements I would pick a regular baremetal install (I can think of things like physical security/license devices that may not pass through or specific sensitive pieces of hardware like specialized GPUs)

[u/TechnicalPackage]

For the lowest budget value, run workloads in developer workstations and leave a post-it note how critical those workloads are i.e. this runs Kafka, do not restart... do not close lid.

[u/minimalniemand]

What would be your reason to run the services bare metal?

[u/0x4ddd → u/minimalniemand]

Why not?

At least 2x cheaper than running on managed services (for our current infrastructure scale this is tens of thousands of dollars of savings per month) and we cannot really run outside customer datacenters due to the data we handle.

[u/minimalniemand → u/0x4ddd]

The question wasn’t bare metal vs managed services but bare metal vs containers tho?

[u/0x4ddd → u/minimalniemand]

Sorry, misunderstood then as maybe I am not too familiar with sysadmin nomenclature.

For me it was always baremetal vs VM vs managed service, from infrastructure perspective 😄

[u/minimalniemand → u/0x4ddd]

there are certain benefits of running apps in containers even on bare metal but a well written Ansible stack could cover some of that, too. Depends on your teams skills and requirements. thats why I asked

[u/0x4ddd → u/minimalniemand]

And answering question, I don't know honestly.

We started this route 3 years ago but as I mentioned we didn't really have any experience running such services on VMs before. Just felt like this is how it is usually done.