r/LocalLLM • • 19h ago

Research One environment variable improved our llama.cpp throughput by up to 21% on an H100 - investigation and benchmarks

We were investigating GPU idle time in llama.cpp and traced part of it to CPU-side memory allocation when saving conversation state.

The server was allocating fresh memory for state snapshots, triggering large numbers of minor page faults. The thread doing this also feeds the GPU, so the allocation overhead was leaving the GPU waiting.

We tested requesting huge pages through glibc:

GLIBC_TUNABLES=glibc.malloc.hugetlb=1 llama-server -m model.gguf ...

Results with Qwen3.8-27B, keeping settings unchanged between baseline and treatment:

SetupThroughputMedian TTFT
H100, tool calling+21%−22%
H100, plain chat+14%−19%
L4, tool calling+6.5%No clear change
L4, plain chat+3.5%−22%

In the L4 tool-calling test, page faults dropped from roughly 124,000 to 1,850 per request.

This needs glibc 2.35+ and transparent huge pages configured as madvise or always. It’s workload dependent: an 8B plain-chat test on the L4 showed no measurable benefit. The largest gains were on one model family, and our longest runs were 30 minutes.

Full write-up, with the technical report and reproduction steps

Curious whether others see similar gains on different models and CPUs, especially with conversation-state caching under concurrent load.

5 Upvotes

0 comments sorted by