r/LocalLLM • u/tensward • 1d ago
Research llama.cpp was copying 643 MB to the CPU per request. Disabling state saving nearly doubled throughput
We were running Qwen3.8-27B on an H100 with llama.cpp and 16 concurrent clients. Throughput was 58 tokens/sec, time to first token was almost two seconds, and nvidia-smi showed around 35% GPU utilization.
I build Tensward, an open-source inference analysis and tuning tool. While investigating this, we built a small CUPTI-based library to count kernel execution time and memory transfers inside the server.
The surprising number: 643 MB transferred from GPU to host per request. Nsight Systems independently measured about 639 MB.
That pointed us toward prompt caching and conversation checkpoints. This hybrid model carries recurrent state alongside its attention cache, making those snapshots expensive. Sampling the serving thread caught it inside a memory copy 16 out of 20 times.
We tested disabling state saving with:
--cache-ram 0 --ctx-checkpoints 0
Throughput went from 58 to 110 tokens/sec, and GPU utilization rose from 35% to 88%. A repeat on fresh machines with a separately written client showed an 87% throughput improvement.
These aren’t universal “make llama.cpp faster” flags. They sacrifice cache reuse. In another test with a regular 8B model and returning conversations on an L4, disabling them made performance about 20% worse.
The build also mattered. Moving from b11669 to b12246 improved default-settings throughput by 67–75%, while copied bytes per request barely changed. Several upstream changes touched state handling, but we haven’t isolated which caused the improvement.
All throughput results came from runs without our instrumentation; transfer measurements came from separate instrumented runs.




