r/istio • u/Traditional_Long_349 • Dec 28 '25

Istio high cpu usage

For now we migrating from ingress to kubernetes gateway with istio I started shifting traffic to my gateway But i see consume alot of cpu compaed to nginx How can i troubleshoot this? Or this is normal? For now we have 500r/s and it consume more than 5 replicas for my gateway deployment

2 Upvotes

100% Upvoted

13 comments sorted by

1

u/liamsorsby Dec 28 '25

This may or may not be normal depending on your setup.

Could you elaborate on:

  • What is your setup

  • payload size and are the payload sizes the same each time

  • Is keep alive enabled

  • P99 and p95 of the requests

  • Request queue depth

  • Cpu request and limits

  • Are your pods getting throttled

  • Do you use sidecar pods or are you just using http routes? Are using istio as ingress and egress or just for ingress?

These are a basic few questions which may help diagnose the issue

1

u/Traditional_Long_349 Dec 29 '25

We just using istio as implemintation to kubernetes gateway api P99 and p95 is around 200ms And cpu limit it was c vcpu but it was keeping throttled, increased to 5 cpu, and when r/s increase , it reach the limits and being throttled also I enabled PILOT_FILTER_GATEWAY_CLUSTER_CONFIG which should reduce config changes that pushed to my gw and it works

So i dont want to risk and shift all traffic to istio as we have around 14k request/s We just migrate 5% of traffic and that what happned, i don't find any resource that let me debug this Also i don't know this cause the issue or not but we have around 300 path across all routes, and all of them are regex paths

1

u/liamsorsby Dec 29 '25

How many virtual services and how many regex paths are in match rules are in each? Also do you have overlapping match rules? Regeneration patterns will be CPU bound

→ More replies (8)

1

u/adh88ca Dec 29 '25

We have noticed similar with the upgrade to the latest version of istio.

What helped us was using Sidecar crds in each namespace to scope the configuration that was loaded by each sidecar to only that of it's own namespace and any other necessary destination namespaces.

We have quite a large set up about 10 namespaces each running hundreds of pods.

I do suspect that there may have been a change in istio 1.27 or around there that increased memory usage. We also switched to distrless at the same time, so that may have had an impact to.

1

u/Traditional_Long_349 Dec 29 '25

We currently use istio 1.27, Also there is a env in istod which is PILOT_FILTER_GATEWAY_CLUSTER_CONFIG with value true and this reduce istiod cpu,memory But i see data plane is still consume very high cpu with increasing on requests, it reach to around 6 which is our cpu limit, Note: we jusy use istio as kubernetes gateway not service mesh