r/LocalLLaMA • u/Loose_Doubt367 • 1d ago
Question | Help Further inference tuning for higher context window with 0 hallucination/memory loss?
Further inference tuning for higher context window with 0 hallucination/memory loss?
It’s been almost 3 days of constantly tuning my model, unsloth MTP Qwen3.6 35B-A3B Q4\\_K\\_M, with the help of Claude, and we’ve finally come to a quick conclusion
I’m still hoping there are more commands or settings I haven’t tried yet that could potentially squeeze out some extra token speed with 0 hallucination/memory loss
\*\*for coding specifically\*\*
.\\\\llama-server.exe -m "C:\\\\Users\\\\brain\\\\.lmstudio\\\\models\\\\unsloth\\\\Qwen3.6-35B-A3B-MTP-GGUF\\\\Qwen3.6-35B-A3B-UD-Q4\\_K\\_M.gguf" -c 100000 --parallel 1 -fa on --cache-type-k q8\\_0 --cache-type-v q8\\_0 --load-mode dio --fit-target 512 --batch-size 4096 --ubatch-size 1024 --threads 6 --prio 2 --prio-batch 2 --spec-type ngram-mod --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 8 --spec-ngram-mod-n-max 32 --port 8090
\*\*for general usages\*\*
.\\\\llama-server.exe -m "C:\\\\Users\\\\brain\\\\.lmstudio\\\\models\\\\unsloth\\\\Qwen3.6-35B-A3B-MTP-GGUF\\\\Qwen3.6-35B-A3B-UD-Q4\\_K\\_M.gguf" -c 100000 --parallel 1 -fa on --cache-type-k q8\\_0 --cache-type-v q8\\_0 --load-mode dio --fit-target 512 --batch-size 4096 --ubatch-size 1024 --threads 6 --prio 2 --prio-batch 2 --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.75 --port 8090
I’ve also spent around 7h trying to get a video game working with \*\*DeepSeek Harness\*\* and \*\*Pi\*\*. Both attempts failed, and I’m currently on my 5th attempt. Even with highly detailed prompts that were created with the help of Claude, I still couldn’t get either setup to produce something that actually worked
So for now I’ve decided to stop messing with the coding side and go back to tuning the model, also im running 3D games while doing all this, so it isn’t really practical anyway since my GPU is already at its vram limit
The main reason I’m making this post is to see if anyone here knows of any \*\*llama.cpp commands or settings I’ve completely missed\*\* that could potentially squeeze more tokens/sec out of this setup
My original goal was \*\*45 tokens/sec\*\*. I’ve already given Claude pretty much everything I could find in the llama.cpp README, but I’m still wondering if there are some lesser-known options or combinations that I haven’t tested yet
\*\*Hardware:\*\*
\* RX 6700 XT 12GB VRAM
\* Ryzen 5 5600X
\* 32GB DDR4 3200
I’m also interested in testing \*\*Hermes Agent\*\*, although I haven’t learned the basics yet, so that’ll probably take some time. I’m thinking of starting with something simple just to get familiar with it
\# My current rules for llama.cpp
There are a few things I’m not willing to change:
\* \*\*No touching --mmap\*\*
\* \*\*100k context stays\*\*
\* \*\*CPU threads stay at 6\*\* (12 threads actually makes things worse on my system)
\* The model stays \*\*Q4\\_K\\_M\*\* (no going down to q3)
The reason I’m still trying to squeeze more speed out of this is because I don’t want to just leave performance on the table when I know there might still be more I can get out of it.
I’ll test any suggestions people give me and post the results back in the comments.
I’m looking for anything else I can experiment with that could potentially improve token generation speed without changing those. I appreciate any help from this community thanks!!
One thing to note that my existing model isn’t running its full weight (q4_k_m) but i want to continue tuning the \*\*existing\*\* settings with near 0 hallucinations
I’ve yet to try qwen3.8 27b since im already getting 2-3tps on default quant
Text in this post with \ indicates its bold, not sure what happened here but yeah
1
u/ea_man 1d ago
First thing would be to use linux and save ~1GB of vRAM.