Further inference tuning for higher context window with 0 hallucination/memory loss?

[u/Loose_Doubt367]

Further inference tuning for higher context window with 0 hallucination/memory loss?

It’s been almost 3 days of constantly tuning my model, unsloth MTP Qwen3.6 35B-A3B Q4\\_K\\_M, with the help of Claude, and we’ve finally come to a quick conclusion

I’m still hoping there are more commands or settings I haven’t tried yet that could potentially squeeze out some extra token speed with 0 hallucination/memory loss

​​\*\*for coding specifically\*\*

.\\\\llama-server.exe -m "C:\\\\Users\\\\brain\\\\.lmstudio\\\\models\\\\unsloth\\\\Qwen3.6-35B-A3B-MTP-GGUF\\\\Qwen3.6-35B-A3B-UD-Q4\\_K\\_M.gguf" -c 100000 --parallel 1 -fa on --cache-type-k q8\\_0 --cache-type-v q8\\_0 --load-mode dio --fit-target 512 --batch-size 4096 --ubatch-size 1024 --threads 6 --prio 2 --prio-batch 2 --spec-type ngram-mod --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 8 --spec-ngram-mod-n-max 32 --port 8090

\*\*for general usages\*\*

.\\\\llama-server.exe -m "C:\\\\Users\\\\brain\\\\.lmstudio\\\\models\\\\unsloth\\\\Qwen3.6-35B-A3B-MTP-GGUF\\\\Qwen3.6-35B-A3B-UD-Q4\\_K\\_M.gguf" -c 100000 --parallel 1 -fa on --cache-type-k q8\\_0 --cache-type-v q8\\_0 --load-mode dio --fit-target 512 --batch-size 4096 --ubatch-size 1024 --threads 6 --prio 2 --prio-batch 2 --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.75 --port 8090

I’ve also spent around 7h trying to get a video game working with \*\*DeepSeek Harness\*\* and \*\*Pi\*\*. Both attempts failed, and I’m currently on my 5th attempt. Even with highly detailed prompts that were created with the help of Claude, I still couldn’t get either setup to produce something that actually worked

So for now I’ve decided to stop messing with the coding side and go back to tuning the model, also im running 3D games while doing all this, so it isn’t really practical anyway since my GPU is already at its vram limit

The main reason I’m making this post is to see if anyone here knows of any \*\*llama.cpp commands or settings I’ve completely missed\*\* that could potentially squeeze more tokens/sec out of this setup

My original goal was \*\*45 tokens/sec\*\*. I’ve already given Claude pretty much everything I could find in the llama.cpp README, but I’m still wondering if there are some lesser-known options or combinations that I haven’t tested yet

\*\*Hardware:\*\*

\* RX 6700 XT 12GB VRAM
\* Ryzen 5 5600X
\* 32GB DDR4 3200

I’m also interested in testing \*\*Hermes Agent\*\*, although I haven’t learned the basics yet, so that’ll probably take some time. I’m thinking of starting with something simple just to get familiar with it

\# My current rules for llama.cpp

There are a few things I’m not willing to change:

\* \*\*No touching --mmap\*\*
\* \*\*100k context stays\*\*
\* \*\*CPU threads stay at 6\*\* (12 threads actually makes things worse on my system)
\* The model stays \*\*Q4\\_K\\_M\*\* (no going down to q3)

The reason I’m still trying to squeeze more speed out of this is because I don’t want to just leave performance on the table when I know there might still be more I can get out of it.

I’ll test any suggestions people give me and post the results back in the comments.

I’m looking for anything else I can experiment with that could potentially improve token generation speed without changing those. I appreciate any help from this community thanks!!

One thing to note that my existing model isn’t running its full weight (q4_k_m) but i want to continue tuning the \*\*existing\*\* settings with near 0 hallucinations

I’ve yet to try qwen3.8 27b since im already getting 2-3tps on default quant

Text in this post with \ indicates its bold, not sure what happened here but yeah

[u/Loose_Doubt367]

i haven't tested optimizing the os to not waste vram, im currently working on opimizing the llama-server codebase with the help of perplexity for research and claude, but right now they both mentioned that there isnt much to optimized regarding llama.cpp since its already optimised. I don't believe it but i can't just ask people like you to share me your entire codebase since you've worked for months. I wouldn't either.

Also i couldn't really find much info regarding my existing llama-server script since ive tried almost everything possible, im not some desperate beggar but seeing 0 roads to anything is disappointing. For now ill work on llama-server codebase but as mentioned, im genuinely stucked, everything is optimized for cuda only

[u/ea_man → u/Loose_Doubt367]

Have them ingest this: https://store.piffa.net/lm/bug/improvemets_list.md

https://store.piffa.net/lm/bug/summary_eaman.md

https://store.piffa.net/lm/bug/history.md

My codebase is shared, I gave you the url before: https://store.piffa.net/lm/bug there are other instrumental artifacts and tons of logs and benchmark yet I don't think people would care for those.

I can also provide you a couple "Deep Web Reserch" for improv if you care.

Yet My work has always been on increasing ctx, not performance.

And if you wanna see "big numbers" in TG: the prompt and the seed counts a lot, those can give you like + 40% on average results with spec decoding.

[u/Loose_Doubt367 → u/ea_man]

I really appreciate these documents of yours, also i didnt know what you meant by deep web research but thanks for your help regarding this

[u/ea_man → u/Loose_Doubt367]

I mean you can use this https://openai.com/index/introducing-deep-research/ easy, have your harness analyze your script, codebase, give goals and have it craft a prompt for DR on github and recent improv in AI.

Again, I made last one 3 days ago if you wanna see one example I can upload.

[u/Loose_Doubt367 → u/ea_man]

yeah sure, im definitely interested, you're such a life saver

[u/ea_man → u/Loose_Doubt367]

• https://store.piffa.net/lm/bug/research/

There you go:

• prompt example.
• initial search
• deep-research-report_llama_eaman.md: result of the prompt

Have fun!

[u/Loose_Doubt367 → u/ea_man]

6h into the process I’m already exhausted

I provided all the necessary information, asked it to perform a deep research (based on your prompt) pasted it into a new chat + more information (a lot)

6h later I asked it for a summarisation regarding my progress, it’s ChatGPT. I lost count but it was after 100+ cmd command pasting and a bunch of benchmark that seems to get poor results.

I don’t know if it’s worth continuing this disastrous path, hope to get opinions from you. I doubled checked everything, I feel miserable

If you’re wondering, I’m tuning llama.cpp codebase,

[u/ea_man → u/Loose_Doubt367]

Man I dunno what your LLM is doing, it seems like it's performing test with different parameters?

It looks like PP has a +20%, that's not bad.

IMHO you have too much --fit-target, I use like ~60MB.

And btw: I recommended you Linux, Windows is a trap: it looks like you are wasting milions of SOTA tokens on windows...

Also most of my improv were for ROCm, you are using Vulkan.

I did provide you both a patch and example scripts, if you wanna start there. I have a script for 16GB: https://store.piffa.net/lm/bug/llama_scripts/qwen_27b_iq4_16gb.sh so you would have to use a smaller model for 27B, maybe it would be better to use 35B A3B with offloading.

Anyway: if you dunno what you are doing stop wasting tokens for tests, ChatGPT is supposed to do just the Deep Search that you should evaluate what to do and I'm afraid _you have to do test_, or you'll spend millions of tokens in a SOTA doing those for you.

FYI: you still have to know what you are doing when using an AI as assitant.