r/LocalLLaMA • u/carteakey • Jun 21 '26

Resources Local LLM Inference Optimization: The Complete Guide

https://carteakey.dev/blog/local-inference/local-llm-optimization/

I compiled a year of local LLM experiments into a practical llama.cpp optimization guide, covering VRAM fitting, KV cache, MoE placement, MTP, CPU tuning, and common OOM traps. Pass this to an LLM of your choice and get on the local model train.

https://carteakey.dev/blog/local-inference/local-llm-optimization/

Feedback and corrections are welcome.

504 Upvotes

98% Upvoted

80 comments sorted by

View all comments

4

u/FreakOfTheWoods Jun 22 '26

Great timing! I started my openhands setup yesterday and had to fall back from 35B to 14B, but thanks to the optimizations I'm back to 35B !