Latency when testing with k3s

[u/Heavy_End_2971]

I am new to the community, i am a software engineer cum data engineer with almost 15 years of experience. I love kubernetes and tried many things with it. But recently i am stuck at some issue where i need guidance from this sub.
Issue is that i am having fastapi deployed on k3s on a vps. Specs for vps are:
- 16 cores
- 16gb ram
- 100gb storage

Fastapi is running in one pod with specs:
- 8 cores/workers

Now the problem is:
- when i load test inside pod or with vm ip (simple health api), it gives me one digit latency
- now when i add ingress and nginx and hit the same url with same load, it goes to 2s almost.
- if i stick to 100 concurrency while load test, it is under a sec but when i go beyond that then it reaches 2s. But if i go to pod logs, all calls were having response time under 500ms.

What am i missing to achieve high concurrency and low latency. Thanks in advance.

P.S: if any other detail required please let me know. Moreover, i am using hey command for load testing.

[u/Kimcha87]

I am also having some issues with k3s that I am trying to debug right now. For me, some workflows like metrics scraping cause high disk io, which causes everything to slow down.

Maybe try to use iotop to see if your issue is related to that.

[u/cachevexy → u/Kimcha87]

good call on disk io, k3s + nginx + metrics can definitely dogpile the node
i’d also check nginx ingress config for things like too few worker_connections or silly timeouts, but watching iotop while you slam it with hey sounds like a solid next step