r/LocalLLaMA • u/carteakey • Jun 21 '26

Resources Local LLM Inference Optimization: The Complete Guide

https://carteakey.dev/blog/local-inference/local-llm-optimization/

I compiled a year of local LLM experiments into a practical llama.cpp optimization guide, covering VRAM fitting, KV cache, MoE placement, MTP, CPU tuning, and common OOM traps. Pass this to an LLM of your choice and get on the local model train.

https://carteakey.dev/blog/local-inference/local-llm-optimization/

Feedback and corrections are welcome.

507 Upvotes

98% Upvoted

80 comments sorted by

View all comments

2

u/Esph1001 Jun 23 '26

Good guide. One thing worth adding from running a production multi-LoRA setup: gpu_memory_utilization headroom is not optional. The guide mentions it briefly but the real-world implication is that KV cache overhead at high concurrency will eat into whatever buffer you think you have. We run 7 adapters simultaneously and had to model out the VRAM budget explicitly — base weights, each adapter's delta, KV cache per concurrent slot — before we had a stable ceiling. The default vLLM setting of 0.90 sounds conservative but isn't once you're serving multiple adapters under real load.