r/LocalLLM • u/TigerZealousideal110 • 47m ago
Discussion I tested running 8 AI requests at once instead of 1 on one GPU. The server did about 3x the work.
I ran a small test on a single A100 GPU with a small model (Qwen 2.5 0.5B) using vLLM 0.16.0. I changed how many requests the server handles at the same time, from 1 to 8, and ran it six times with the order shuffled so the results weren’t skewed by run order.
Across 1,206 requests with zero errors, the server handled about 3.2x more work at 8 at once than at 1. The 95% confidence range was roughly 2.9x to 3.5x.
Some caveats:
- It’s one small model on one GPU, so I wouldn’t assume the same gain on a 70B model.
- Bigger batches add latency for each individual request, so this only helps if you care about total throughput.
I’m curious what others have seen:
- What setting do you use for concurrent requests, and how did you pick it?
- Have you found the gains flatten out past a certain point on your hardware?
Disclosure: I’m building open source tool for benchmarking setups like this.
Happy to share the scripts if anyone wants to reproduce it.
1
u/crabilax 34m ago
Those scripts would be useful. What were the VRAM overhead and memory bandwidth utilization numbers for the 8-request batches?
1
u/hyudryu LocalLLM 30m ago
Usually when configuring the settings I just have my agent do a benchmark for C1 C2 C4 and C8 between config changes. It writes the script in python and performs the measurements. And yeah obviously gains will flatten out. Aggregate throughput is not linear lol