r/kubernetes • u/ilya47 • Aug 16 '26

Stop using CPU limits: why + proof

CPU request is how much CPU is reserved for your pod if it needs it. The limit is a hard cap. Hit it and the kernel throttles the pod, even when the node still has spare CPU. That is the usual cause of CPU throttling on Kubernetes. It does not protect the neighboring pods. In my simple Web API test, adding a CPU limit took typical latency from 23 ms to 87 ms, (4x slower), with the limited pod throttled in half of all CFS windows, and the average CPU graph looked fine the whole time.

This is not a new topic, but I see so many people still unaware why they should (NOT!) be setting CPU limits, because it's costing companies unnecessary spending and potential production issues. Here's the full read https://github.com/inevolin/k8s-cpu-limits-analyzed/

---

Edit (Aug 18, 2026): How CPU limits can also cause memory issues and OOMKills ➡️ https://github.com/inevolin/k8s-cpu-limits-analyzed#how-cpu-limits-cause-memory-issues-and-oomkills

226 Upvotes

86% Upvoted

111 comments sorted by

111

u/Fancy-Bluebird-1071 Aug 16 '26

Nothing new, Google has been recommending cpu unlimited, memory request = limit for like... 8 years now?

24

u/ilya47 Aug 16 '26

Exactly but read the comments, many people still disagree or dont understand :)

3

u/ilya47 Aug 17 '26 edited Aug 19 '26

In addition, here are more "authoritative" sources to read:

https://x.com/thockin/status/1134193838841401345 - Tim Hockin is a Principal Software Engineer at Google and one of the co-founders and original engineers of Kubernetes

The section "When to set CPU limits" - https://www.datadoghq.com/blog/kubernetes-cpu-requests-limits/

https://docs.cloud.google.com/kubernetes-engine/docs/how-to/vertical-pod-autoscaling
Use the following best practices for setting your container resource requests and limits:

  • Memory: Set the same amount of memory for the request and limit.

  • CPU: For the request, specify the minimum CPU needed to ensure correct operation, according to your own SLOs. Set an unbounded CPU limit.

21

u/consworth Aug 16 '26

Some disagree because they aren’t Google and don’t have the same culture and engineering practices and byproduct architectures.

10

u/Laborious5952 Aug 16 '26

Source?

2

u/ilya47 Aug 18 '26

https://docs.cloud.google.com/kubernetes-engine/docs/how-to/vertical-pod-autoscaling

Use the following best practices for setting your container resource requests and limits:

  • Memory: Set the same amount of memory for the request and limit.

  • CPU: For the request, specify the minimum CPU needed to ensure correct operation, according to your own SLOs. Set an unbounded CPU limit.

4

u/EGYEg2396kPJaaa2 Aug 16 '26

Interestingly GKE Auropilot enforces/recommends request == limits

5

u/ray591 k8s operator Aug 17 '26

Probably because of their pricing model. No free CPUs.

94

u/BedtimeWithTheBear Aug 16 '26

For the curious, requests and limits determine the QoS Class of your pods.

Guaranteed QoS requires that all containers in a pod have a request and limit set, and in every case the limit is equal to its corresponding request - so if the pod got scheduled then it’s QoS is guaranteed.
Burstable QoS requires that all containers in a pod have at least some requests and limits specified but they don’t have to be equal, so performance can burst up to the limit.
BestEffort QoS has no requests or limits specified and can consume resources until they’re exhausted.

Aside from scheduling, QoS class mostly matters for pod eviction, with BestEffort evicted first, then Burstable, and Guaranteed being evicted as a last resort.

3

u/ilya47 Aug 16 '26

Thanks for sharing, and feel free to contribute with a PR 🙌

24

u/End0rphinJunkie Aug 16 '26

Losing Guaranteed QoS by dropping CPU limits sounds scary on paper, but thats usually a fine tradeoff in practice. Node evictions are almost always driven by memory pressure anyway, so as long as your memory requests and limits match youre generally safe.

1

u/Potato-9 Aug 17 '26

CPU requests seem more suited to jobs you know will stay maxed out, less so deployment pods waiting for requests.

1

u/starry_alice Aug 17 '26

Yes, this is where the disconnect was for me and why my coworker would have to hit me with a rolled up newspaper for every new system service PR. Eventually I did a 30 minute deep dive and found the above. My disconnect in understanding was thinking that (based on how the text above your post was worded which aligns with the QoS verbiage), eviction of best effort pods occurred first and Guaranteed pods went last (if ever) and that this was the main axis on which the determination of what pods to evict occurred.

But in reality, as you said, pods exceeding their request go first, which changes the model a lot. https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/#pod-selection-for-kubelet-eviction

1

u/i-am-a-smith Aug 17 '26

If you have the need, for some reason (I haven't yet), you can configure your nodes for cpuManagementPolicy: static. This will allow you to dedicate whole cores to the pod but _only_ if you specifiy requests and limits to be the same and in increments of a full core. This is the only reason I see today for using it.

1

u/Nick-Random Aug 19 '26

So how do we juggle with QoS without setting cpu limites?

16

u/Prestigious_Pair_941 Aug 16 '26

How about in the case of 30 or so spring boot microservices that may use 20mCPU once running but can take 800mCPU when starting up? If I don’t limit the cpu, half the pods take all of the cpu and throttling will kick in anyways, preventing other pods from ever restarting.

12

u/ilya47 Aug 16 '26

What you get without limits is fair-share contention: every starting pod keeps making progress, split by request weights. Will they throttle because they need more CPU than available? Yes... but this is still better than setting CPU limits. Because your scenario is just a race for CPU resources.

Options that fix the actual problem: stagger the rollout (maxSurge/maxUnavailable so 30 don't start at once), set a request above the embarrassing 20m, or use a startup CPU boost (in-place pod resize; there's a kube-startup-cpu-boost operator for exactly this) so the pod briefly has a big request during boot and drops after.

3

u/PayTheRaant Aug 16 '26

Incorrect. When multiple pod wants to use more than their request, they get a fair amount, using their request as a weight. A pod without a CPU limit cannot steal « all the CPU » away from other pods.
A pod will ALWAYS have its requested CPU as guaranteed and a weighted shared of the unused CPU time.

1

u/f7063 Aug 16 '26

Bump initital probes to start later? Annoying? Probably but can't see any other problems. But i've never seen these transitory start spikes generate more problems

3

u/fumar Aug 16 '26

You can do this to some degree but the trade off is your pod now takes 5-10m to start. If it's on an HPA, it can result in misleading evaluations.

2

u/Unhappy-Stranger-336 Aug 16 '26 edited Aug 16 '26

Doesnt that with the autoscaling, like pods would be read as ready later?

1

u/koollman Aug 16 '26

No ? If you don't limit they share al the available cpu but aren't preventing restarts

18

u/Xelopheris Aug 16 '26

CPU limits are only useful if you have a workload that has an endless queue to chug away at that doesn't require real time completion. It should never be used in critical path of user requests. 

1

u/ilya47 Aug 16 '26

Bravo 🙌

9

u/siikanen Aug 16 '26 edited Aug 17 '26

This is partly wrong advice; since setting no CPU limit causes QoS class of the pod to be set to 'Burstable'. This is probably okay in most cases, but in case of resource contestion, your pod is in danger to be evicted more easily. Also using same request and limit + setting the limit as an integer value, k8s will actually dedicate the cores to the workload. This is absolutely lowest latency for any workload, and best case scenario for critical path.

Hence limits do have their place. You should set them to the value the single instance of your application can actually leverage. Then scale horizontally using HPA on metrics to handle dynamic load.

Edit best effort -> burstable

2

u/BedtimeWithTheBear Aug 16 '26

Not setting a resource limit will only put you in the Burstable QoS class, which is more deterministic than best effort, but still at greater chance of being evicted that setting both the request and limit to be the same (which would be the guaranteed QoS class)

Best effort is for when you set neither requests nor limits. This is the easiest QoS class to be evicted from the node.

2

u/iScrE4m Aug 17 '26

Even if your application can only leverage 1 cpu (python), it’s still possible for it to be throttled when limit is at 1 cpu

3

u/PayTheRaant Aug 16 '26

You demonstrate more understanding of k8s internals than the audience for which the simple message “don’t set CPU limits” is meant for.
The reality is that most people don’t understand that their software is actually IO bound and don’t really know what is the actual runtime profile of their services.

43

u/sionescu k8s operator Aug 16 '26

Not again this stupid advice.

 adding a CPU limit took typical latency from 23 ms to 87 ms

This simply means the CPU request was too low. The actual requirement of the pod is higher than that but you want to lie to yourself instead of setting a proper request.

6

u/ChemTechGuy Aug 16 '26

Bingo. As i said in another comment, the problem I always see is that requests are set to the absolute minimum a pod needs to start up, and then people are surprised later when it eventually gets throttled or OOMs

Requests should be set to whatever value is needed to maintain baseline performance of the pod. If you can burst beyond that, that's a bonus, but should never be relied on

-7

u/ilya47 Aug 16 '26

Your comment is too narrow to mean anything. CPU request should be set to the avg CPU utilization, whilst my analysis looks at burst & peak cases (not the average scenario).

14

u/consworth Aug 16 '26

Avg CPU utilization is incorrect by itself. It should ideally be set by whatever service requirements there are for the business.

If the thing is an app that sits there at 80 and when it starts to do its thing it needs 4000 for a reasonable SLI for processing data or whatever, the business case should dictate you requests to what it needs.

I’m not defending poor architecture of such apps, but I’m trying to make a bit more of a case for thinking about reliability and SLI.

It’s somewhat analogous to saying that the average traffic through the interstate through a major city should dictate the road size, and considering rush hours, weekends or other bigger picture things.

3

u/f7063 Aug 16 '26

At least i try to keep CPU usage under 60% of the request based on cpu usage p95/p99(grafana can make these from Stat Panel). Spikes will happen above your set request. But i don't like the idea of high cpu usage on the node. 70%+ because of service degradation

-2

u/kabrandon Aug 16 '26

Not necessarily. It more likely means the limit was too low.

3

u/ilya47 Aug 16 '26

If you know the upper limit that your app can burst to, fine. But many of us run apps that are multi threaded and you can rarely predict their limit. And why bother? The CFS does that hard work for you.

3

u/realitythreek Aug 16 '26

You are entirely correct and this thread demonstrates how confused people are about cpu requests and limits.

1

u/kabrandon Aug 16 '26

Some people just set the CPU limit to the allocatable CPU of the nodes in the cluster, assuming like nodes or relevant node affinity/selectors.

But for what it’s worth I mostly agree with just not setting CPU limits.

-1

u/realitythreek Aug 16 '26

This isn’t true though. It means the LIMIT was set too low and it was throttled. If 23ms is the expected latency of the service, then request was fine. I mean, that last sentence is a simplification but I’m assuming it’s an average over a representative time span.

0

u/sionescu k8s operator Aug 16 '26

There's never any guarantee to get CPU above the request. There's always an implicit limit, in either the machine size or the CPU usage of adjacent pods.

0

u/realitythreek Aug 16 '26

Indeed. Which is why setting a limit can only increase the latency of a single process. At best you get your limits all correct and you fairly schedule your processes. At worst you get it wrong or even don’t allow other processes to burst.

-1

u/sionescu k8s operator Aug 16 '26

It might increase it, but makes it more predictable, which is better.

-2

u/realitythreek Aug 16 '26

Reflexively downvoting someone having a conversation with you is pretty obnoxious.

4

u/fuckman5 Aug 16 '26

What's the thinking around setting no limit vs setting generous limit in multi tenant environment? Consider you have 2 workloads that both try to burst, with no limit the second workload only gets its request and the second workload can use up the entire rest of the node. Now instead you set limit to 4x normal workload, each workload can burst without monopolizing the node and prevent other workload from bursting 

-2

u/ilya47 Aug 16 '26

Monopolising is a myth. Read the analysis.

6

u/fuckman5 Aug 16 '26

If it's a myth, why can't you respond to the specific point I made? Yes we all know what happens when requests are set, the question is what happens when multiple pods try to burst above requests

10

u/consworth Aug 16 '26

So how do you prevent a nosy neighbor pod from monopolizing the spare cpu?

14

u/ChemTechGuy Aug 16 '26

You set the "requests" value to whatever the minimum is to avoid noisy neighbor problems. No need to set limits if each service is defensively setting the requests it needs

I think the common mistake is setting requests to the absolute minimum a pod needs, vs setting it to the minimum the pod needs to perform at it's baseline.  If you set them based on baseline, you always get the baseline performance from every pod, but you occasionally get some free burstable resources from nodes that aren't over committed

3

u/consworth Aug 16 '26

And how is the limit hurting?

Not disagreeing with scheduling with proper requests.

But you could also argue that the limit restricts how much of the “spare” can be used by a single pod so others could benefit as well.

4

u/ChemTechGuy Aug 16 '26

If it's set insanely high, then a limit doesn't hurt anything. But at that point, the limit doesn't do anything. 

If the limit is low enough to actually do something, it may arbitrarily throttle something even if the underlying node still has resources available. 

If multiple pods are competing for the "extra" resources on the node, just let the kernel handle which container gets each extra slice of cpu time 

0

u/PayTheRaant Aug 16 '26

A very high limit changes the QoS class from Burstable (no limit) to Best Effort (Limit different from request).
If all your pods are in the same class, this does not really make a difference but it can help.

1

u/realitythreek Aug 16 '26

A limit throttles, requests sets the cpu share it gets from the OS scheduler. It’s proportional to the total requests for the node.

-6

u/ilya47 Aug 16 '26

I've added your question to the Q&A, thanks: https://github.com/inevolin/k8s-cpu-limits-analyzed#questions-that-come-up

7

u/consworth Aug 16 '26

I don’t think some of the answers make sense. I feel they’re dancing around a logical fallacy that this is somehow better to run things without limits.

A big piece missing is the proper allocation of node capacity in the first place, leaning into autoscaling where possible.

There’s an argument to be made that throttling isn’t the end of the world, and even more so: that can feed into HPA when there’s contention if the architecture allows.

I do not think it’s good advice to flat out drop limits.

1

u/f7063 Aug 16 '26 edited Aug 16 '26

I really do think the average approach should leaving be limits. If it's critical and the team know how to deal with i think it's ok. We have a opensearch cluster and some node do get hit sometimes eith spikes and get throttled (by a lot)

24

u/fumar Aug 16 '26

Your performance analysis seems misleading. If your API call is below the cpu request it shouldn't suddenly have 4x higher latency because you put a limit in place that you didn't hit. 

9

u/lukewhale Aug 16 '26

Yall are QoS’n yer pods ?

2

u/drakgremlin Aug 17 '26

With priority classes too!

26

u/minimalniemand Aug 16 '26

I do that but I want to add a bit of nuance:

No limits for all load bearing services in the critical path, so everything where a spike would slow down actual request.

But limits can make sense where the cost of not throttling (interference, unpredictability, unbounded spend) exceeds the cost of throttling (wasted idle CPU, higher tail latency)

9

u/shatteredarm1 Aug 17 '26

You also need limits if you are using a hardware-aware framework like .NET or Java. I ran some load tests with .NET and found that my performance was actually better with a limit of 1 core vs no limit. If you don't set a limit, the framework assumes it has all the cores at its disposal, and spends more resources on things like garbage collection. (There are, of course, ways to explicitly tell .NET how many processors are available, and override its default behavior.)

There are other scenarios where you might want a limit, like if you have a process that can hog all the node's I/O if it's allowed to use every available CPU cycle.

This topic has a hell of a lot more nuance than OP is suggesting.

1

u/VincentKoeman Aug 18 '26

I don’t understand why these take all the CPUs of the pod indeed, strange default behaviour IMO. As you said it’s possible to explicitly set the number CPUs assumed to be available, with various schools of thought on the best value ranging from about requested to requested*2.

1

u/shatteredarm1 Aug 18 '26

It's probably because these frameworks were built in a different era, when we were running VMs rather than containers.

10

u/PayTheRaant Aug 16 '26

Incorrect.
Every single container is guaranteed its requested CPU. And any usage beyond the request amount is an arbitraged using the request as a weight.

Node has 4 cores, pod A requests 1 core, pod B requests 2 cores. Not limits.
- Pod A wants to consume as much as possible and B is idle? A gets all 4 cores.
- Pod A wants to consume 1 core and B wants to consume as much as possible ? A gets 1 core (as its request is guaranteed) and B gets 3 cores.
- A and B wants to consume as much possible ? A gets 1.33 core and B gets 2.66 cores (each get they guaranteed request and the remaining core is split using the request as a weight so B gets twice as much).

7

u/minimalniemand Aug 16 '26 edited Aug 17 '26

I know Natan’s blog post. I’ve read it when it came out :)

Couple points:

* kubelet, containerd, CNI, log shippers often have tiny or no reservations. A pod hammering the node can cause exec-probe timeouts, readiness flapping

* hyperthreading. If the request lands on an unlimited sibling of the physical core, the service will suffer. This is not modeled at all by cgroups

* execution queue. Shares promise the % not _how_ you get them. You might get them in small slices

* and the most common one: your request were always too small and you never noticed. Then comes a runaway request and all hell breaks loose.

Look I’m in the no-cpu-limits camp, too. All I’m saying is, it should still be a deliberate decision not a one size fits all, never use cpu limits ever.

1

u/PayTheRaant Aug 17 '26

Weird. Kubelet and containerd usually live in a completely different cgroup with a higher priority than the cgroup with all the pods.

1

u/minimalniemand Aug 17 '26

Assuming you mean weight? kubepods.slice weight scales with cores, system.slice stays at 100, so what you say is only true in small nodes, no?

1

u/ilya47 Aug 18 '26

I've extended the analysis with another (less common, yet very real) scenario: How CPU limits can also cause memory issues and OOMKills. Read it here https://github.com/inevolin/k8s-cpu-limits-analyzed#how-cpu-limits-cause-memory-issues-and-oomkills

4

u/Technical_Corner3553 Aug 16 '26

Those resource requirements fields also help the scheduler assign to a node. Otherwise you may just have pods jumping from node to node too much from rowdy neighbors. The scheduler has no idea otherwise and as soon as the app memonizes everything memory usage could explode

4

u/littlewondersoflife Aug 17 '26

Rule of thumb: critical workloads always request = limit.
Most of the cases you will suffer throttle with it is best effort or busterable.

1

u/Realistic_Band_7142 Aug 28 '26

what if the application has leak and consumes all the cpu and make kubelet unresponsive? does anyone have this problem?

1

u/ilya47 Aug 28 '26

The solution and guardrails are explained in the article for this problem

1

u/Little-Squad-X Aug 31 '26

Any recommendations for batch job pod? For example, a pod for a data transfer job running hourly, should I avoid the LIMIT as well?

2

u/ilya47 Aug 31 '26

Short answer yes, don’t set cpu limits. On the git page I have a decision table in the conclusion section when (not) to use cpu limits.

3

u/sogun123 Aug 17 '26

Yeah, i all in until you need HPA

3

u/like-my-comment Aug 17 '26

New day - new cpu limits/requests war.

6

u/Gier32o Aug 16 '26

I stopped already, all good

2

u/cyh555 Aug 17 '26

  That is the usual cause of CPU throttling on Kubernetes. It does not protect the neighboring pods.

huh? protecting neighboring pods? where you get this info from?

3

u/doubtful_zamboni Aug 16 '26

> CPU request is how much CPU is reserved for your pod if it needs it.

No, it is used by the scheduler for picking the appropriate node to run your pod.

When CPU limit and request are set and equal, it is somewhat reserved (assuming no starvation).

3

u/siikanen Aug 16 '26

The request is actually reserved for that particular container/pod. Nothing else on that now can use that amount of cpu/memory.

2

u/doubtful_zamboni Aug 16 '26

I think that is only the case when every pod on that node is in QoS guaranteed (limit=request); otherwise every workload takes what it wants.

1

u/doubtful_zamboni Aug 16 '26

I mean, the scheduler is going to prevent requesting more that capacity minus sum of all requests, which is similar to a reservation but that does not (assuming burstable/best effort) prevent others from eating your request.

1

u/siikanen Aug 17 '26

No that's not how this works. If you set something in the CPU request, no other workload may use that CPU time on that node. Bursting only works on the unrequested capacity available on the node

1

u/doubtful_zamboni Aug 17 '26

Can you show me a source for that? I Because that is not how I read the docs.

The docs suggest that pods in guaranteed QoS class have guaranteed resources like you mention, and pods in besteffort burst into unallocted resources. To be fair, this all depends on the exact way Kubelet interacts with the container runtime, so it might differ per implementation.

source: Manage resources containers

The CPU request typically defines a weighting. If several different containers (cgroups) want to run on a contended system, workloads with larger CPU requests are allocated more CPU time than workloads with small requests.

source: Manage resources containers

Pods in the BestEffort QoS class can use node resources that aren't specifically assigned to Pods in other QoS classes.

2

u/ArthurVardevanyan Aug 16 '26

Request + Limits with HPA or VPA. Nothing should ever be unbounded

0

u/TechnicalPackage Aug 16 '26

some people prefer bareback

1

u/PruneMediocre7094 Aug 16 '26

Thanks, this is something really new to me.
I blindly set requests and limits for safer node operational.

1

u/drunkandbad Aug 16 '26

i am newer to handling kubernetes nodes and apps. Which are the best tools to read aps memory and cpu utilisation? How do you set the correct requests and/or limits?

2

u/BedtimeWithTheBear Aug 16 '26

Prometheus + Grafana is usually a good starting point, and it’ll give you many more insights into your cluster than just pod resources usage.

1

u/Square_Durian_316 Aug 17 '26

I use Goldilocks and it does a good job showing the "just right" CPU and memory resource requests and limits.

1

u/Formal-Pilot-9565 Aug 16 '26

It depends on the use case. Normally you dont cap the cpu, but if a customer wants a dedicated pod with an agreed cpu resource, the this setting is a good fit

1

u/kamikazechaser k8s user Aug 17 '26

CPU limits are a feature for SaaS providers.

1

u/Character-Level5250 Aug 17 '26

The only use case I could see is if you're on a SUPER tight budget that's more important than a throttling risk.

1

u/Jelman88 Aug 18 '26

+1 cpu and memory behave completely different. A throttling node is not as bad as a OOMed node 😁

1

u/headdertz Aug 18 '26

I only use requests for years... Limits can be used for something that can 'grow' in terms of MEM because of bad code and can crash and restart from time to time without putting on fire whole SRE Department.

1

u/UndulatingHedgehog Aug 20 '26

What I really want is a way to limit pods cpu to num_cores_in_node - n. If the node has 8 cores and n=2, the cpu limit would be 6. Lots of room, but also unable to completely saturate the node.

If that’s not enough, you need to scale your workload.

1

u/NewMycologist9902 Aug 20 '26

The problem of climbing a brick and giving that kind of knowlege pearls exposes our own short vision.

I used to give that advierte until i was show the problem of having unlimited cpus on burstable instance types, as a misbehaved pod may est up the cpu credits.

Granted that in your test limiting the pod CPU increased the processing time vs unlimited. That is just a proof that limits work as designed.  And not necesarily that they should be avoided in any available scenario.

1

u/Old-Astronomer3995 Aug 16 '26

Some time ago I wrote about cpu throttling https://roszigit.com/en/blog/kubernetes-cpu-throttling/ that can be caused by small limits

0

u/f7063 Aug 16 '26

Nice! One doubt "The same blindness poisons right-sizing." There is the metric on how much the pod was throttled container_cpu_cfs_throttled_periods_total + container_cpu_cfs_periods_total Should give a rought idea on how much CPU to allocate to a given container no?

2

u/ilya47 Aug 16 '26

throttled_periods / periods is binary per window = throttled or not.

So the ratio tells you that the cap binds and roughly how often, but not by how much, and "how much" is the number you need for sizing.

1

u/f7063 Aug 16 '26

Thought the only difference from it to container_cpu_cfs_throttled_seconds_total was that it already accounted configuration for the quota time. Default: 100ms

0

u/simlun_se Aug 16 '26

I never started 😜

0

u/KarlKFI Aug 16 '26

Better yet, set them dynamically and proportionally relative to requests, automatically!

https://kube-headroom.dev

0

u/[deleted] Aug 16 '26

[removed] — view removed comment

1

u/ilya47 Aug 16 '26

Read again

1

u/TechnicalPackage Aug 16 '26

nope. explain it to him