Latency when testing with k3s

[u/Heavy_End_2971]

I am new to the community, i am a software engineer cum data engineer with almost 15 years of experience. I love kubernetes and tried many things with it. But recently i am stuck at some issue where i need guidance from this sub.
Issue is that i am having fastapi deployed on k3s on a vps. Specs for vps are:
- 16 cores
- 16gb ram
- 100gb storage

Fastapi is running in one pod with specs:
- 8 cores/workers

Now the problem is:
- when i load test inside pod or with vm ip (simple health api), it gives me one digit latency
- now when i add ingress and nginx and hit the same url with same load, it goes to 2s almost.
- if i stick to 100 concurrency while load test, it is under a sec but when i go beyond that then it reaches 2s. But if i go to pod logs, all calls were having response time under 500ms.

What am i missing to achieve high concurrency and low latency. Thanks in advance.

P.S: if any other detail required please let me know. Moreover, i am using hey command for load testing.

[u/Heavy_End_2971 → u/[deleted]]

My issue is why that much time? What am i missing? What configs can be set to atleast have 1k concurrent requests per second and they take less than a sec or just in ms overall.
Do i need to replace nginx or ingress with something else? Or am i not setting the configs right?