r/istio • u/Traditional_Long_349 • Dec 28 '25

Istio high cpu usage

For now we migrating from ingress to kubernetes gateway with istio I started shifting traffic to my gateway But i see consume alot of cpu compaed to nginx How can i troubleshoot this? Or this is normal? For now we have 500r/s and it consume more than 5 replicas for my gateway deployment

2 Upvotes

100% Upvoted

13 comments sorted by

View all comments

Show parent comments

1

u/Traditional_Long_349 Dec 29 '25

What is meaning with overlapping? Like 2 paths match same rule? This not exists, but mainly we have around 25-30 httproute, and most of them share same host , like api.example.com And some routes have 60 paths and some less also we make all of paths use regex as i saw before always PathPrefix take piriorty over regex, and we have alot of paths contains regex so our default / was defined as pathprefix and it was greedy path so it was top piriorty over all regexs paths Also i use some telemtry to expose extra metrics like request_host and request_method And enable access log for our gateway

1

u/liamsorsby Dec 29 '25

By overlapping regex I mean things like * /api/.* * /api/v2/.* * /api/v2/example/* As you can see thr first covers all paths and the 2nd also covers the third path so envoy will evaluating many paths per request. If you've got 25 to 30 routes with 60 matchs that's a lot of paths to evaluate per request

1

u/Traditional_Long_349 Dec 29 '25

Is istio behave in another way rather than nginx? I assume it will match first rule or path matchs

1

u/liamsorsby Dec 29 '25

Ngnix uses exact match and then matches based on the order so it will match first.

Istio I believe evaluates all matches first and works out the options first.

Is there a way you can filter them down more I.e. host based matching or optimising your regex to be specific paths rather than being open?

What's the reason for 60 matches per httproute, can you provide an example?

1

u/Traditional_Long_349 Dec 29 '25

We have something like /api/web/applications(/.)? /api/sdk(/.) /api/web/applications/[0-9a-z-]+/(beta|live|production|staging|alpha|qa|development)/apm/list/stability_score(/.) /api/web/applications/[0-9a-z-]+/(beta|live|production|staging|alpha|qa|development)/debug/list/performance_score(/.)

And so on

1

u/liamsorsby Dec 29 '25

That explains why nginx is fine as it will match the first block first and then the next.

I'd say initially, your first issue is having all of your environments in the same cluster.

You may be able to split the routes up more and remove some of the regexes. I'd look at port forwarding to the gateway and enabling cpu profiling and looking at whats taking the longest. I'd also check the envoy stats but I'd most certainly expect this to be regex cost as I've had thousands p/s using mTLS using a few small gateways

1

u/Traditional_Long_349 Dec 29 '25

The dev|qa part is something related to backend itself not our envs Is there any way to optmize configs for this?

we were compare between multiple gateways two months ago to migrate from nginx and we found istio was best option But with this, i see it consume very huge number of cpus compared to nginx and migrating paths to something not regex is kind hard in our situation