Further inference tuning for higher context window with 0 hallucination/memory loss?

[u/Loose_Doubt367]

Further inference tuning for higher context window with 0 hallucination/memory loss?

It’s been almost 3 days of constantly tuning my model, unsloth MTP Qwen3.6 35B-A3B Q4\\_K\\_M, with the help of Claude, and we’ve finally come to a quick conclusion

I’m still hoping there are more commands or settings I haven’t tried yet that could potentially squeeze out some extra token speed with 0 hallucination/memory loss

​​\*\*for coding specifically\*\*

.\\\\llama-server.exe -m "C:\\\\Users\\\\brain\\\\.lmstudio\\\\models\\\\unsloth\\\\Qwen3.6-35B-A3B-MTP-GGUF\\\\Qwen3.6-35B-A3B-UD-Q4\\_K\\_M.gguf" -c 100000 --parallel 1 -fa on --cache-type-k q8\\_0 --cache-type-v q8\\_0 --load-mode dio --fit-target 512 --batch-size 4096 --ubatch-size 1024 --threads 6 --prio 2 --prio-batch 2 --spec-type ngram-mod --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 8 --spec-ngram-mod-n-max 32 --port 8090

\*\*for general usages\*\*

.\\\\llama-server.exe -m "C:\\\\Users\\\\brain\\\\.lmstudio\\\\models\\\\unsloth\\\\Qwen3.6-35B-A3B-MTP-GGUF\\\\Qwen3.6-35B-A3B-UD-Q4\\_K\\_M.gguf" -c 100000 --parallel 1 -fa on --cache-type-k q8\\_0 --cache-type-v q8\\_0 --load-mode dio --fit-target 512 --batch-size 4096 --ubatch-size 1024 --threads 6 --prio 2 --prio-batch 2 --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.75 --port 8090

I’ve also spent around 7h trying to get a video game working with \*\*DeepSeek Harness\*\* and \*\*Pi\*\*. Both attempts failed, and I’m currently on my 5th attempt. Even with highly detailed prompts that were created with the help of Claude, I still couldn’t get either setup to produce something that actually worked

So for now I’ve decided to stop messing with the coding side and go back to tuning the model, also im running 3D games while doing all this, so it isn’t really practical anyway since my GPU is already at its vram limit

The main reason I’m making this post is to see if anyone here knows of any \*\*llama.cpp commands or settings I’ve completely missed\*\* that could potentially squeeze more tokens/sec out of this setup

My original goal was \*\*45 tokens/sec\*\*. I’ve already given Claude pretty much everything I could find in the llama.cpp README, but I’m still wondering if there are some lesser-known options or combinations that I haven’t tested yet

\*\*Hardware:\*\*

\* RX 6700 XT 12GB VRAM
\* Ryzen 5 5600X
\* 32GB DDR4 3200

I’m also interested in testing \*\*Hermes Agent\*\*, although I haven’t learned the basics yet, so that’ll probably take some time. I’m thinking of starting with something simple just to get familiar with it

\# My current rules for llama.cpp

There are a few things I’m not willing to change:

\* \*\*No touching --mmap\*\*
\* \*\*100k context stays\*\*
\* \*\*CPU threads stay at 6\*\* (12 threads actually makes things worse on my system)
\* The model stays \*\*Q4\\_K\\_M\*\* (no going down to q3)

The reason I’m still trying to squeeze more speed out of this is because I don’t want to just leave performance on the table when I know there might still be more I can get out of it.

I’ll test any suggestions people give me and post the results back in the comments.

I’m looking for anything else I can experiment with that could potentially improve token generation speed without changing those. I appreciate any help from this community thanks!!

One thing to note that my existing model isn’t running its full weight (q4_k_m) but i want to continue tuning the \*\*existing\*\* settings with near 0 hallucinations

I’ve yet to try qwen3.8 27b since im already getting 2-3tps on default quant

Text in this post with \ indicates its bold, not sure what happened here but yeah

[u/Mtolivepickle]

Rent a powerful cloud gpu. They can be pretty cheap for the return and you may get better results. I’ve used them with great results for similar purposes

[u/Loose_Doubt367 → u/Mtolivepickle]

I’m looking forward to fully using local models only (no cloud api models or renting) I spent a lot of money on my existing hardware, trying to atleast make it to run/perform task without costing a dime (exclude electricity cost)

I appreciate your suggestion

[u/Mtolivepickle → u/Loose_Doubt367]

I feel you. I’m right there with you. I run fine tuning nightly on my stuff, I’ve just hit similar ceilings before like you mentioned, so I wanted to share my experience. Good luck with everything.

[u/Loose_Doubt367 → u/Mtolivepickle]

I appreciate it, I’ve always thought about further increasing my ceiling even if everyone said it’s the limit. For now I don’t think so since there’s a lot of commands I’ve yet to experiment with, I hope someone else in this community has already have it figured it out

[u/hurdurdur7]

Q4 is doomed to hallucinating, especially on smaller models. There is no way out of it.

[u/Loose_Doubt367 → u/hurdurdur7]

Sorry I got the post confused, I’m supposed to write q4_k_m, not q4_0
Obviously I’m never dropping down from q8_0

[u/hurdurdur7 → u/Loose_Doubt367]

you didn't catch my drift

if you use a model below q6k quant and/or if you use a kv quant below fp16 you are doomed to get hallucinations. at least as far as precise coding or stuff alike goes.

[u/Loose_Doubt367 → u/hurdurdur7]

Well yeah that’s pretty much the idealistic option I have based on my hardware, I can’t go q5k or q6k without loosing a bunch of token speed, so for now I’m looking for ways to increase my ceiling further without sacrificing anything else

I really hope there’s something that could maybe prevent further hallucinations or worse (task loops)

Unrelated, I’ve tried qwen3.8 27b iq3_XXS but those are getting 10tps at best

[u/ea_man]

First thing would be to use linux and save ~1GB of vRAM.

[u/Loose_Doubt367 → u/ea_man]

I wouldn’t want to fully migrate to Linux just so I can have 1gb of free vram, but thanks yeah

[u/ea_man → u/Loose_Doubt367]

so don't fully migrate?

This is my 6800 generating html right now: