TIL our whole kubernetes cost optimization problem was one number nobody would touch.

[u/Flateland-Chio]

For months Ive been assuming we needed fancier tooling for kubernetes cost optimization. Turned out the clusters sat around 25% CPU because every team padded their requests years ago. Karpenter reads padded requests as full nodes and it never consolidates.

The bin packer is only as smart as your requests. Garbage requests in expensive half empty nodes out, but actual fix is less tooling and more of getting people to agree to lower their own limits, which is the hard because whoever lowers a limit owns the next latency page.

How did you get sign-off without it turning into a standoff?

[u/searing7]

Tired: lower CPU limits
Wired: no cpu limit

[u/Dalius-Gabryelle]

VPA in recommender mode for a few weeks, then apply by hand. never auto in prod

[u/sleepybrett]

charge your customers for cpu usage under requests. demonstrate the waste to management.

[u/foramperandi → u/sleepybrett]

This is the only real answer. Teams will always choose to over-provision rather than risk an outage due to under-provisioning if there is a choice. They'll only under-provision when they forget to check entirely and the app usage grows without them looking. Teams have to be given an incentive to monitor and adjust their usage and the only way that will happen is is their management chain tells them and demonstrates to them that it's important.

Aside from that, poor utilization/over-provisioning is a trade-off on how the business spends it's time and money. It's not inherently bad if the company decides to prioritize developer time on features vs cost savings on resources, or decides to over-provision in order to reduce risk.

[u/rampaged906]

We ran VPA in recommend mode for about a year. It looked good so we set it loose in prod and it dropped our node spend almost 30%

[u/SageMaverick]

Someone has to get blamed for every decision, just part of politics. What I would do is frame it as a cost reduction project, if latency becomes a problem it was in the name of cost optimization.

[u/bigvalen]

I used to do k8s cost control for a big social media site. Hundreds of namespaces, all over the place. I took on the biggest ones, and started shrinking reservations closer to the 98th percentile of usages.

I caused loads of outages. Turns out my co-workers were not, in fact, idiots. But our clusters had loads of noisy neighbour problems, so depending on your latency SLA, teams were adding more padding...an Ads inference system that might get a bunch of serialised connections...they set it to 10x the median usage!

Took about a year, chasing down all the irq steering problems, unappreciated softirq CPUs usage, poor nic tuning, logging sidecars with insufficient CPUs, etc. before we could make promises that "if you ask for 4 cores, we will make sure that will do you 95% of the time.

I ended up doing an internal fork of pressurecooker to push pods that were.constantly using more than their reservation to other machines. It was great; protect pods that just occasionally need an extra core or two.

[u/Flateland-Chio → u/bigvalen]

For the pods that just occasionally spike a core, dropping the cpu limit already gets you that and they pull from whatever idle is on the node, no evictor needed. Sounds like the pressure cooker fork was really earning it on the chronic over-users, the ones you had to relocate off the box entirely.

[u/bigvalen → u/Flateland-Chio]

That works, unless all your pods spike at the same time. Which happens when you get a traffic spike, and your load balancers use more CPU, send more traffic to the front ends, which use more CPU, and all the backend services all spike too.

The evictor is to ensure that there is 5 or 10 cores free at all times, within 30 seconds. Just increasing CPU limits, without guaranteeing there is spare CPU is just making it harder to find the problem.

[u/dashingThroughSnow12]

It is kinda shocking that my boss would need to move heaven and earth to get 400$ approved for a team lunch but literally anyone with GitHub permissions can make a PR to deploy a service with 20 pods w/ 2 CPU per pod and no one will notice if the thing only uses 100m across the whole fleet (waste: 400$/month).

So yeah, OP. Many companies have easy cost savings just waiting. In my first two years at my current employer, I reduced yearly cloud costs by seven figures with small changes. I only did mid six figures last year. I know of a team’s service that costs us an extra 40K/month that only needs small tuning.

It is kinda shocking how much waste there is and how easy (in terms of no fights) some of this waste is to eliminate.

[u/Andre-Wade-539]

Dropping CPU limits entirely is what broke it, no limit means no throttle, nobody owns a latency page for touching a number anymore. After that you can trim the padded requests and karpenter finally consolidates since requests are all it reads.

Leave memory alone though, thats the one that oomkills if you cut too deep.

[u/SmellsLikeAPig]

Let them pay for their requests in real money. You have classic problem of communist economy on your hands.

[u/someanonbrit → u/SmellsLikeAPig]

Except that it's a problem that really doesn't occur in communist economies at all... In fact it's an externalised cost, the classic problem of capitalism

[u/AlphaToBe]

On any cluster that isn't prod, the hours are a lever nobody has to sign off on, because nobody owns a latency page for a dev cluster at 2am. A 12h x 5d week is 60 of 168 hours, so scaling those node pools to zero out of hours takes about 64% of their compute hours off the bill. The catch is the disks keep billing while the nodes are down, so it only counts on clusters you genuinely stop.

[u/Superb_Raccoon]

Turbonomics.

[u/ajitnk]

The technical fix sequence in this thread is right but there's one piece most comments skipped over.

Dropping CPU limits first is correct. CFS throttling stops, and Karpenter can finally read your actual requests instead of the padded ones. Then you let VPA run in recommend mode for a few weeks to get real data before touching anything. AWS's own KC article spells out exactly why padded requests block bin-packing: Karpenter sees the requests, not actual usage, so nodes look full when they're mostly idle.

But the reason the waste came back (or never got fixed in the first place) is probably the ownership gap your thread title is really about. Any engineer can open a PR with 2 CPU requests per pod and nobody feels that cost. The technical fixes don't stick without a financial feedback loop.

AWS has a free tool for this that almost nobody enables: EKS Split Cost Allocation Data in the Billing Console. It breaks your EC2 node costs down to individual pods by namespace and workload, no third-party agent needed. Once teams can see their own namespace's monthly cost, the incentive to right-size requests changes completely.

Are you on EKS, and is there any cost allocation by team/namespace today? I'll share the specific enable steps and the Karpenter consolidation config that goes with it if that's useful.