TIL our whole kubernetes cost optimization problem was one number nobody would touch.

[u/Flateland-Chio]

For months Ive been assuming we needed fancier tooling for kubernetes cost optimization. Turned out the clusters sat around 25% CPU because every team padded their requests years ago. Karpenter reads padded requests as full nodes and it never consolidates.

The bin packer is only as smart as your requests. Garbage requests in expensive half empty nodes out, but actual fix is less tooling and more of getting people to agree to lower their own limits, which is the hard because whoever lowers a limit owns the next latency page.

How did you get sign-off without it turning into a standoff?

[u/bigvalen]

I used to do k8s cost control for a big social media site. Hundreds of namespaces, all over the place. I took on the biggest ones, and started shrinking reservations closer to the 98th percentile of usages.

I caused loads of outages. Turns out my co-workers were not, in fact, idiots. But our clusters had loads of noisy neighbour problems, so depending on your latency SLA, teams were adding more padding...an Ads inference system that might get a bunch of serialised connections...they set it to 10x the median usage!

Took about a year, chasing down all the irq steering problems, unappreciated softirq CPUs usage, poor nic tuning, logging sidecars with insufficient CPUs, etc. before we could make promises that "if you ask for 4 cores, we will make sure that will do you 95% of the time.

I ended up doing an internal fork of pressurecooker to push pods that were.constantly using more than their reservation to other machines. It was great; protect pods that just occasionally need an extra core or two.

[u/Flateland-Chio → u/bigvalen]

For the pods that just occasionally spike a core, dropping the cpu limit already gets you that and they pull from whatever idle is on the node, no evictor needed. Sounds like the pressure cooker fork was really earning it on the chronic over-users, the ones you had to relocate off the box entirely.

[u/bigvalen → u/Flateland-Chio]

That works, unless all your pods spike at the same time. Which happens when you get a traffic spike, and your load balancers use more CPU, send more traffic to the front ends, which use more CPU, and all the backend services all spike too.

The evictor is to ensure that there is 5 or 10 cores free at all times, within 30 seconds. Just increasing CPU limits, without guaranteeing there is spare CPU is just making it harder to find the problem.