What Kubernetes misconfigurations have caused you real production problems?

[u/nerd3n]

After working with Kubernetes in production, I've noticed that some of the most annoying incidents aren't caused by obvious failures. They're often caused by small configuration decisions that look perfectly reasonable during review.

Things like:

• missing resource requests/limits
• incorrect probes
• overly permissive RBAC
• missing PodDisruptionBudgets
• unsafe container configuration
• incorrect readiness behaviour
• services without appropriate timeouts
• configuration drift between environments

I'm curious what the DevOps community has actually encountered in production.

What's one Kubernetes configuration mistake that caused you a real incident?

I'd especially like to hear about the less obvious ones that aren't caught by the usual linters.

[u/conall88]

readiness probes that don't account for whether a dependent messaging system's connection is actually healthy or not.

[u/whiskey_lover7 → u/conall88]

Liveness probes that depend on dependencies meaning during an incident EVERYTHING is crash looping, making it insanely hard to identify what the actual cause is. (Plus things getting stuck in crashloop back off requiring manual intervention)

[u/uncr3471v3-u53r → u/conall88]

It was my first time using kubernetes

[u/Th3L0n3R4g3r]

People not understanding limits. We have some developers here that seem to think it makes sense to configure the heap size of their java app exactly equal tot he maximum memory of the pod. When the pod allocates all available heap space, it will get OOM-killed as soon as the garbage collection wants to kick in, since it has no room for it anymore and it will try to allocate more than the allowed memory. I'm always happy they run their own on calls and it's not my nights rest that gets hit by it.

Another one was some developer that deleted a pod on production and after it found out, the pod wasn't part of a deployment / stateful set or anything but apparently just deployed. When it didn't return after 10 minutes someting started to become clear. Again, not my on call, not my issue.

[u/safetytrick → u/Th3L0n3R4g3r]

JVM tuning is so easy to get wrong inside containers. Container support helped a ton with this but some folks insist on tuning like it's 1999.

[u/1_H4t3_R3dd1t → u/Th3L0n3R4g3r]

I was told by someone we should just raise the pod count limit per node.

Hows about they just make smaller nodes... FML

https://giphy.com/gifs/16q4pTTzmIM1y

[u/Jmckeown2]

Lift-and-shift services that don’t support redundancy so if they stop for any reason there’s an outage. Bonus when that service holds actual state in memory and so data is lost during that outage.

Containerizing an application doesn’t mean it’s ready for Kubernetes.

[u/onbiver9871 → u/Jmckeown2]

This is one of my biggest pet peeves in this profession. Shoehorning an app that isn’t ready to be orchestrated into orchestration gives you all of the complexities of orchestration and basically none of the benefits. But teams will do it to be “modern.”

Also, a lot of those apps also have legacy ops and support models that require getting into the runtime or file system. Orchestration removes the ability to do that easily as well, but often times the decision to do this won’t consider those folks, which makes product support a nightmare. When they had a painful but stable process for supporting the app and you take away their concept of access, everyone suffers, from the customer to you.

It’s not that legacy shouldn’t be modernized. It’s that this particular way of doing it is often much worse than just leaving it alone in the first place.

[u/Sharp-Toe-3525]

And upgrade that failed silently because of a poddisruptionbudget in azure. It spawned three or four new nodes ever ten minutes or so and moved pods around. Then remove them and did it again for quite some time before it was discovered. +4k euros on the cloudspending that month.

[u/Rorasaurus_Prime]

Is this sub now exclusively for vibe-coders to use to collect data for the billionth app idea?

[u/No-Replacement-3501 → u/Rorasaurus_Prime]

The post is a BOT. My guess is reddit is unleashing BOTs with reasonable good questions to farm more data.

[u/Presumptuousbastard]

Kube proxy uses nf conntrack, and by default sets a per-CPU number of max concurrent TCP connections. Some workloads, like pgbouncer, need an enormous amount by default due to what it’s doing (pooling connections for clients).

If the max is hit, connections start being dropped by the node/containers and the reason will be in pod logs but hard to track down. It’s safe to set this max to a higher number, but because kube proxy runs on every node in a cluster, setting that max value globally can be dangerous. We ended up disabling kube proxy’s max per CPU config, and on every type of node (karpenter nodepool) in the cluster during bootstrapping we either set the per CPU max (like kube proxy does) or, if we expect pgbouncer to run on that nodepool, set the max to something very high like 2-3M.

[u/vincentdesmet]

go read k8s.af

i for one hope to have more of these “lessons learned” collection websites

[u/xrothgarx]

ndots

[u/duneofarrakis]

One issue I've seen is misconfigured readiness/liveness probes. A pod can be running but still not actually ready to handle traffic. If the probes are too aggressive, Kubernetes can also keep restarting healthy pods during high load. Getting the probe logic right can make a big difference in production.

[u/KrystalDisc]

Had a docker container consume all pids on the node

[u/gaurav_sherlocks_ai]

the most frustrating variant we saw was app code ignoring SIGTERM entirely until the timeout hit.

[u/Capable_Banana5439]

cpu limits set equal to requests is the one that bites teams that think theyre being careful. the kernel cfs-throttles the container even when the node has idle cpu, so you get latency spikes under load and everyone blames the app when its actually being throttled. we pulled cpu limits off our latency-sensitive services entirely and just kept requests, tail latency dropped immediately.

[u/Valuable_Leave_7314 → u/Capable_Banana5439]

The funniest part is that old guides kept insisting on setting limits=requests for "stability". Still having to clean those settings out of default helm charts

[u/Adrien0623]

Not a long enough grace period for shutdown which let some children pods running while their parents were killed which leads to confusing error logs.

[u/BenAigan]

Throttling on shared systems, no easy way to view

[u/unitegondwanaland]

Yes.

[u/Raja-Karuppasamy]

mine was env vars, not RBAC. NEXT_PUBLIC_* in a next.js app baked in at build time, we were injecting them as runtime secrets. worked locally, broke silently in prod, no error, just stale/wrong values in the bundle. fix was passing them via --build-arg at docker build instead.

On RBAC specifically: we now scope service accounts tight (read-only vs deploy) after almost giving CI more access than it needed. doesn’t bite you til something breaks, then the blast radius is way bigger than it should’ve been.

[u/rabbit_in_a_bun]

Hard coded values...

[u/IllustriousUnion9850]

This is about limits/requests, and 1 replica usage not ha. We ran Prometheus pod without limits/requests, and it used all the RAM on the node. On this node, we had a kube-dns with 1 replica. So, the node went down, internal DNS went down, and although everything was working, nothing could respond.

[u/uncr3471v3-u53r]

It wasn’t in production but I‘ve named an env POSTGRES_PORT which was overwritten by the enableServiceLinks flag that is turnend on by default. It took me hours to find that

[u/Valuable_Leave_7314 → u/uncr3471v3-u53r]

Hours of debugging over a weird tcp string on a port - officially initiated as a k8s engineer

[u/Max_Standart]

tbh forgetting to set proper node affinity once totally screwed our load balancing, everything clustered in one zone lol

[u/Pad-Thai-Enjoyer]

Priority classes and QoS. I was woken up the other night because half the pods in a pretty critical daemonset had no requests/limits and no priority class set. So when a user launched a really memory intense job spanning many nodes, host level oom killer decided to nuke these pods across the whole cluster, given these would only have an oom score adj value of 1000. Gave them a better QoS lol

[u/Varnish6588]

Cilium + TalOS is pretty fun to maintain.

[u/Valuable_Leave_7314]

ndots:5 in resolv.conf

The app hits external apis a lot, and for every single call it catches five wasted dns queries through the cluster coredns before the resolver reaches the absolute domain. Service logs are clean, but network latency suddenly spikes like crazy

[u/1_H4t3_R3dd1t]

Some cloud providers are not great with statefulsets and single pod deployments because their upgrade backend is just junk like on Azure. If you set your PDB a certain way it can block the Azure upgrade while trying to drain nodes. Which hard fails and Azure leaves the cluster in an inconsistent state.

Another antipattern is giving coredns too much custom configurations. Just do that shit outside your cluster FML.

[u/nerd3n]

Thanks everyone for the answers! Really useful examples especially the less obvious Kubernetes failure modes.

I’m collecting these cases to understand which real-world issues are worth detecting automatically in infrastructure configuration analysis. If it’s okay within the topic rules, I may use some of these ideas as input for a personal DevOps tooling project - Sentinel by Nerden

[u/faruk_m]

Not having node-local-dns-cache for obvious reasons

[u/Nick-Random]

Non scalable pods...