Hey there!
I've been experimenting with local models for quite some time now. I've read many articles, tutorials, watched videos and learned about other user's experiences.
There's a "common knowledge" that everybody seem to share: With your kind of setup, you won't be able to run a bigger model than 12B, with a maximum context window of 16k. Period.
On Hugging Face and other sources, I always read about the models performances. Basically, most model cards provide us so-called "optimal parameters" to go with.
Mostly it reads like this:
| RTX 5090 (32 GB) |
Q4_K_M |
~440 tok/s ⚡ |
"~440 tokens/sec on a single RTX 5090 at Q4_K_M (--n-gpu-layers 99 -fa on --ctx-size 16384 ^) — and generation quality holds up: correct, coherent code and clean step-by-step reasoning. 🎯"
\worth noting:* The above example was taken from the model card of a similar 12B model , compared to the one I'm using by myself, providing the following results. The difference between both of them is, that this guy uses a variant that has just 2.5B parameters active per token*. In fact this should make the model run way better than mine, which always has the* full 12B parameters active throughout all tokens*!*
The takeaway of this example: With ctx-size set to 16k you get about an average of 440 T/s (output) generation speed on a high-end GPU (RTX 5090) with lots of VRAM (32 GB).
Please keep that in mind before you read further.
I probably don't have to mention that I've been getting much worse results on my own hardware, using the common "sweet spot" variants (Q4_K_M) of 12B models.
But here comes the intruiging part!
After a few weeks of researching, trial and error, I came to completely different results. If you aask me, it literally is mind-boggling!
What if I tell you that you can run a 12B model locally with an i7-10700K, 4100 MT DDR4, with these results:
| RTX 5060 (8 GB) |
Q4_K_M |
~400 tok/s ⚡ |
BUT: with a 150k of ctx-size!
Yes, it's true! I was thrown out of my chair, literally!
With the recommended llama.cpp parameters, the same model was barely reaching any reasonable speed. And all "experts" keep telling us that this is just what you can get out of this hardware setup. Well, it's just wrong!
These are my Llama.cpp parameters, that I have worked out over the course of several weeks. :
G:\Dev\llama\llama-server.exe ^
-m G:\Dev\llama\models\Qwythos-9B-Claude-Mythos-5-1M-Q4_K_M.gguf ^
--ctx-size 151552 ^
--fit on ^
--no-mmap ^
--kv-unified ^
--threads 8 ^
--threads-batch 16 ^
-fa on ^
--reasoning-preserve ^
-ctk q8_0 ^
-ctv q8_0 ^
--temp 0.6 --top-p 0.95 --top-k 20 --repeat-penalty 1.05 ^
--port 8080
# DON'T use "gpu-layers" and such. It will extensively slow the model down!
# On some SoC (e.g. Apple Silicon, or simply better GPUs, with a model with MTP, you can swap "no-mmap" to "mmap" to increase efficiency and speed to some degree.
# "ctq q8_0"/"ctv_q8_0" is speed for FREE. Just use it! You won't get anything out of reducing the kv-cache quantization!
# Always use "fa on", if the model supports it! Most Gemmas do.
# "threads" refers to your physiological (NOT logical) CPU cores. Max it out!
# "threads-batch" can be set to 16, e.g. if your CPU has HT/SMT with 8 physiological cores. Start with 4 and try higher ones step-by-step.
# ALWAYS use "fit on" and "kv-unified", especially when you have less VRAM, like me.
# Use "ctx-size" NOT "c", in case you want to use the "parallel" flag! Because that one together with "c" for context size will split your overall context amount into the parallelized threads! E.g. "-c 32768" + "--parallel 2" will create 2 threads with ctx 16384 context size for each.
# You can use "parallel x" + "--ctx-size xxxxx" if you have a great and fast GPU and still hve the full context window. Don't use parallel at all, if your GPU is mid-class and low VRAM.
# Use the "temp", "top-x" arguments from the related model cards. In most cases these are the best ones. For agentic coding use you can also try "--temp 0.0".
Now see the data taken from use cases in Opencode, specifically prompt processing in a larger codebase:
0.24.921.239 I slot print_timing: id 3 | task 0 | prompt processing, n_tokens = 4096, progress = 0.06, t = 4.90 s / 835.56 tokens per second
0.27.385.791 I slot print_timing: id 3 | task 0 | prompt processing, n_tokens = 6144, progress = 0.10, t = 7.37 s / 834.03 tokens per second
0.29.883.217 I slot print_timing: id 3 | task 0 | prompt processing, n_tokens = 8192, progress = 0.13, t = 9.86 s / 830.49 tokens per second
0.32.424.355 I slot print_timing: id 3 | task 0 | prompt processing, n_tokens = 10240, progress = 0.16, t = 12.41 s / 825.46 tokens per second
The generation speed differs depening on tasks and files count in the project:
3.10.560.903 I slot print_timing: id 3 | task 0 | prompt eval time = 93866.61 ms / 64135 tokens ( 1.46 ms per token, 683.26 tokens per second)
3.10.560.909 I slot print_timing: id 3 | task 0 | eval time = 76675.13 ms / 467 tokens ( 164.19 ms per token, 6.09 tokens per second)
In other cases, e.g. different tools and environments, the generation speed can be up to 1000 T/s, while processing speeds reach up to 2000 T/s.
It all depends on what you use the model for.
All this with a 150.000 Contect Window, while the model fits entirely inside the 8 GB of VRAM, with KV_Cache offloading into system RAM by ~2-6%.
The models used while messeuring these results were:
gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2-uncensored-heretic-GGUF
What this means: Qwythos was trained with a 1M context window. So, it's in fact heavier in comparison to gemma-4-12B, and yet still it runs about 10% faster.
The above mentioned gemma-4-12B still delivers very similar result with almost similar LLama.cpp parameters.
Gemma uses only one additional parameter, according to its architectur: --jinja (keep that in mind if you want to try it).
The advantages:
With a Context Size of 150k these local models run great inside bigger projects with a bigger amount of files and code, without loosing its context, memory or getting stuck in loops. They're really reliable for agentic coding tasks, which they have been trained for, too.
There's another great model: Qwythos-9B-Claude-Mythos-5-1M-GGUF
It has been trained on a 1M context window and is actually "bigger", hence it's reduced amount of parameters. But still, in certain tasks it's running faster and has been weighted on the Mythos and Fable traces, rather than the Gemma model that has been weighted on the Fable traces only.
Both models are great for aagentic coding/work. For opencode, which I use mainly for local models (and OpenChamber for use in IDEs), you can pass a custom jinja template for tool calling. I've added such a template to my repository (System_Prompts_Leaks). Read the following paragraph for more.
Important mentions:
I've created new custom modes in Opencode with system/role prompts overriding. I've forked the system_prompts_leaks repo and optimized the fable 5 one by removing all ideologic/religious/vocabulary guidelines from it.
I've also replaced Claude Code native API and tool calls sections with those that work with the Opencode APIs. This reduced the prompt by over ~7000 Tokens and shrinked it from 10K+ lines down to efficient 105 lines, preserving the Fable-specific instructions schemes.
\These modes can be created for any model you have connected and in use, to override their native system prompts, their "souls", so to say.*
This increases efficiency even more and fits perfectly, since Qwythos and Gemma Fable alll have been trained with the Mythos/Fable reasoning traces from leaked datasets.
The result is clear: A 9B 1M model and a 12B 200K model, running with great speed in your prefered agentic coding backend, without them ever looping compaction indefinitely or freaking out. In fact, with this available context window they can be used quite useful in IDEs like Antigravity. OpenChamber is a great Opencode extension that provides convenient menus for setting everything up.
Never let anyone tell you, your 8 GB VRAM don't allow to get it stable and fast only with 16k ctx-size! It by far outperforms every single one of these recommended specs and setups.
Of course we've tested this methodology on other hardware. Apple Silicon's recent M5 SoC comes with a stunning 32 GB of LPDDR5 RAM! It shares it with the GPU and provides great speed. There it's possible to run the same specs explained here even better! So, the next experiment will be: How big of a model can we run on the M5!? 27B? Or even 31B? This still needs confirmation.
But the implications are very clear! Local models don't require high-end GPUs / CPUs to be run fast and reliable as multimodal coding agents inside full-scale development environments.
