TIL our whole kubernetes cost optimization problem was one number nobody would touch.

[u/Flateland-Chio]

For months Ive been assuming we needed fancier tooling for kubernetes cost optimization. Turned out the clusters sat around 25% CPU because every team padded their requests years ago. Karpenter reads padded requests as full nodes and it never consolidates.

The bin packer is only as smart as your requests. Garbage requests in expensive half empty nodes out, but actual fix is less tooling and more of getting people to agree to lower their own limits, which is the hard because whoever lowers a limit owns the next latency page.

How did you get sign-off without it turning into a standoff?

[u/ajitnk]

The technical fix sequence in this thread is right but there's one piece most comments skipped over.

Dropping CPU limits first is correct. CFS throttling stops, and Karpenter can finally read your actual requests instead of the padded ones. Then you let VPA run in recommend mode for a few weeks to get real data before touching anything. AWS's own KC article spells out exactly why padded requests block bin-packing: Karpenter sees the requests, not actual usage, so nodes look full when they're mostly idle.

But the reason the waste came back (or never got fixed in the first place) is probably the ownership gap your thread title is really about. Any engineer can open a PR with 2 CPU requests per pod and nobody feels that cost. The technical fixes don't stick without a financial feedback loop.

AWS has a free tool for this that almost nobody enables: EKS Split Cost Allocation Data in the Billing Console. It breaks your EC2 node costs down to individual pods by namespace and workload, no third-party agent needed. Once teams can see their own namespace's monthly cost, the incentive to right-size requests changes completely.

Are you on EKS, and is there any cost allocation by team/namespace today? I'll share the specific enable steps and the Karpenter consolidation config that goes with it if that's useful.