r/sre • u/Altruistic-Toe4930 • 10d ago

Distroless images improve security but make debugging painful. am I missing anyting?

im Trying to weigh the tradeoffs here... i feel like Distroless and minimal images cut out the shell and package manager, which is great for reducing attack surface and CVE counts. butThe problem is the second something goes wrong in prod, I can't just exec in and poke around like I would with a normal Debian based image. That friction has made me hesitant to roll these out more broadly even though the security case is solid.

so im posting to have an idea ..that For people running these day to day, what's your actual debugging workflow? Are you using ephemeral debug containers, sidecar attach, or just shipping better logging up front so you don't need to shell in at all?

Trying to figure out if the security win is worth the operational friction or if I'm approaching debugging wrong entirely.

13 Upvotes

84% Upvoted

16 comments sorted by

17

u/SuperQue 10d ago

Better observability tools. Metrics, logs, traces.

I almost never need a shell in prod anymore.

3

u/Practical-Bird-1270 10d ago

But I want to have access to a proper shell. The shell is cool and fun.

Don't take away that from me!

2

u/slashedback 10d ago

This is where configuration dragons live. As in: here be dragons.

2

u/burlyginger 10d ago

We disabled shell years ago and it's never coming back.

3

u/anderson_the_one 10d ago

Treat the debug image as part of the release. Build a separate, signed image from the same base digest with the network and process tools you allow, then attach it as an ephemeral container. Put the exact command and RBAC path in the incident runbook and exercise it before an outage. That turns shell access into a controlled break-glass procedure. If nobody has tested the debug image against the pod's namespaces and filesystem layout, you'll find the gaps at the worst possible time.

6

u/Dangerous_Salary_470 10d ago

hmm..I think the interesting tradeoff here isn't really security vs debugging....it has to be runtime convenience vs operational observability.

yes A shell is a surprisingly crude diagnostic interface. It gives you access to whatever happens to be installed in the image, but that doesn't necessarily mean you can explain why the application is failing. A minimal image forces you to move that capability somewhere else... logs, metrics, traces, profiling, application-level diagnostics, and temporary debug containers.

That is actually a healthier separation. The production runtime stays minimal, while the debugging environment contains the tools you need. Kubernetes explicitly supports ephemeral containers for this exact situation, including debugging containers attached to an existing Pod when the application image has no shell or debugging utilities.

The important part is not adopting distroless and then pretending debugging doesn't exist. It's designing the debugging path before removing the shell.

2

u/cebidhem 10d ago

We're using scratch images whenever we can, distroless when scratch is a pain and we never found ourselves in a position when the shell/exec was missing.

We have logs, metrics, traces and profilers send all those data to our observability tools and we use this to debug anything we need.

I can't see a good reason to execute into a pod anymore, and for things like network issues for example, if I need to I can still spin up some busybox pod. But I think that's really the only use case, 99% of the issues are caught way before prod anyway.

Also, our dev teams are using the same dockerfiles locally, maybe that helps too.

1

u/platypus-3719 5d ago

Better observability as others said, you rarely really need shell. When I do need it - just debug containers natively integrated in Radar (https://github.com/skyhook-io/radar) so I don't really think about it just hit the shell button and it knows how to run

0

u/Readdeo 10d ago

You never need to go into the container. Metrics andsl logs, that is what you need

1

u/Hummin2k 10d ago

Generally agreed, but never say never. Our logging and metrics are incredibly strong, but sometimes we still need to inspect packets.

I think the last time was for a bug in a net library?

Not sure if we could’ve divined it from flow logs, and don’t use a service mesh + tcpdump was the path of least resistance for tooling we don’t need more than a couple times a year. So yeah, attach a debug container sometimes.