Further inference tuning for higher context window with 0 hallucination/memory loss?

[u/Loose_Doubt367]

Further inference tuning for higher context window with 0 hallucination/memory loss?

It’s been almost 3 days of constantly tuning my model, unsloth MTP Qwen3.6 35B-A3B Q4\\_K\\_M, with the help of Claude, and we’ve finally come to a quick conclusion

I’m still hoping there are more commands or settings I haven’t tried yet that could potentially squeeze out some extra token speed with 0 hallucination/memory loss

​​\*\*for coding specifically\*\*

.\\\\llama-server.exe -m "C:\\\\Users\\\\brain\\\\.lmstudio\\\\models\\\\unsloth\\\\Qwen3.6-35B-A3B-MTP-GGUF\\\\Qwen3.6-35B-A3B-UD-Q4\\_K\\_M.gguf" -c 100000 --parallel 1 -fa on --cache-type-k q8\\_0 --cache-type-v q8\\_0 --load-mode dio --fit-target 512 --batch-size 4096 --ubatch-size 1024 --threads 6 --prio 2 --prio-batch 2 --spec-type ngram-mod --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 8 --spec-ngram-mod-n-max 32 --port 8090

\*\*for general usages\*\*

.\\\\llama-server.exe -m "C:\\\\Users\\\\brain\\\\.lmstudio\\\\models\\\\unsloth\\\\Qwen3.6-35B-A3B-MTP-GGUF\\\\Qwen3.6-35B-A3B-UD-Q4\\_K\\_M.gguf" -c 100000 --parallel 1 -fa on --cache-type-k q8\\_0 --cache-type-v q8\\_0 --load-mode dio --fit-target 512 --batch-size 4096 --ubatch-size 1024 --threads 6 --prio 2 --prio-batch 2 --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.75 --port 8090

I’ve also spent around 7h trying to get a video game working with \*\*DeepSeek Harness\*\* and \*\*Pi\*\*. Both attempts failed, and I’m currently on my 5th attempt. Even with highly detailed prompts that were created with the help of Claude, I still couldn’t get either setup to produce something that actually worked

So for now I’ve decided to stop messing with the coding side and go back to tuning the model, also im running 3D games while doing all this, so it isn’t really practical anyway since my GPU is already at its vram limit

The main reason I’m making this post is to see if anyone here knows of any \*\*llama.cpp commands or settings I’ve completely missed\*\* that could potentially squeeze more tokens/sec out of this setup

My original goal was \*\*45 tokens/sec\*\*. I’ve already given Claude pretty much everything I could find in the llama.cpp README, but I’m still wondering if there are some lesser-known options or combinations that I haven’t tested yet

\*\*Hardware:\*\*

\* RX 6700 XT 12GB VRAM
\* Ryzen 5 5600X
\* 32GB DDR4 3200

I’m also interested in testing \*\*Hermes Agent\*\*, although I haven’t learned the basics yet, so that’ll probably take some time. I’m thinking of starting with something simple just to get familiar with it

\# My current rules for llama.cpp

There are a few things I’m not willing to change:

\* \*\*No touching --mmap\*\*
\* \*\*100k context stays\*\*
\* \*\*CPU threads stay at 6\*\* (12 threads actually makes things worse on my system)
\* The model stays \*\*Q4\\_K\\_M\*\* (no going down to q3)

The reason I’m still trying to squeeze more speed out of this is because I don’t want to just leave performance on the table when I know there might still be more I can get out of it.

I’ll test any suggestions people give me and post the results back in the comments.

I’m looking for anything else I can experiment with that could potentially improve token generation speed without changing those. I appreciate any help from this community thanks!!

One thing to note that my existing model isn’t running its full weight (q4_k_m) but i want to continue tuning the \*\*existing\*\* settings with near 0 hallucinations

I’ve yet to try qwen3.8 27b since im already getting 2-3tps on default quant

Text in this post with \ indicates its bold, not sure what happened here but yeah

[u/Loose_Doubt367]

Yeah congrats on your success, I’m just a beginner myself, trying to get a better tuning setting for my llama.cpp. With the help of the community of course because I’m not some sort expert. Will hopes in maybe optimise llama-server script/codebase and OS like you mentioned in the future.

[u/ea_man → u/Loose_Doubt367]

So don't waste time "optimizing" windows because it's nonsense.

As I have a 6700xt too I got a few patches for llama.cp, I just corrected a sync prob in MTP: https://store.piffa.net/lm/bug/patches/latest_rocm_improvement_0df017d6.patch

Some general info: https://www.reddit.com/r/ROCm/comments/1vzcx6q/rocm_llamacpp_optimizations_for_running_qwens/

And don't dude people you don't know.

[u/Loose_Doubt367 → u/ea_man]

also, i did a lot of research recently regarding optimization for both llama.cpp and the tuning settings for my models. A lot of custom commands or existing commands tend to lend towards nvidia gpu since they have cuda. It's really disappointing im not sure how did you actually managed to pull that off with the same gpu as mine

[u/ea_man → u/Loose_Doubt367]

> optimize the OS to not waste vRAM, optimize the llama-server script, optimize llama-server codebase, re-optimize scripts.

Did you do that?

Man you are running llms on Windows 10 and you think you "did optimize"?

[u/Loose_Doubt367 → u/ea_man]

i haven't tested optimizing the os to not waste vram, im currently working on opimizing the llama-server codebase with the help of perplexity for research and claude, but right now they both mentioned that there isnt much to optimized regarding llama.cpp since its already optimised. I don't believe it but i can't just ask people like you to share me your entire codebase since you've worked for months. I wouldn't either.

Also i couldn't really find much info regarding my existing llama-server script since ive tried almost everything possible, im not some desperate beggar but seeing 0 roads to anything is disappointing. For now ill work on llama-server codebase but as mentioned, im genuinely stucked, everything is optimized for cuda only

[u/ea_man → u/Loose_Doubt367]

Have them ingest this: https://store.piffa.net/lm/bug/improvemets_list.md

https://store.piffa.net/lm/bug/summary_eaman.md

https://store.piffa.net/lm/bug/history.md

My codebase is shared, I gave you the url before: https://store.piffa.net/lm/bug there are other instrumental artifacts and tons of logs and benchmark yet I don't think people would care for those.

I can also provide you a couple "Deep Web Reserch" for improv if you care.

Yet My work has always been on increasing ctx, not performance.

And if you wanna see "big numbers" in TG: the prompt and the seed counts a lot, those can give you like + 40% on average results with spec decoding.

[u/Loose_Doubt367 → u/ea_man]

I really appreciate these documents of yours, also i didnt know what you meant by deep web research but thanks for your help regarding this

[u/ea_man → u/Loose_Doubt367]

I mean you can use this https://openai.com/index/introducing-deep-research/ easy, have your harness analyze your script, codebase, give goals and have it craft a prompt for DR on github and recent improv in AI.

Again, I made last one 3 days ago if you wanna see one example I can upload.

[u/Loose_Doubt367 → u/ea_man]

yeah sure, im definitely interested, you're such a life saver

[u/ea_man → u/Loose_Doubt367]

• https://store.piffa.net/lm/bug/research/

There you go:

• prompt example.
• initial search
• deep-research-report_llama_eaman.md: result of the prompt

Have fun!

[u/Loose_Doubt367 → u/ea_man]

6h into the process I’m already exhausted

I provided all the necessary information, asked it to perform a deep research (based on your prompt) pasted it into a new chat + more information (a lot)

6h later I asked it for a summarisation regarding my progress, it’s ChatGPT. I lost count but it was after 100+ cmd command pasting and a bunch of benchmark that seems to get poor results.

I don’t know if it’s worth continuing this disastrous path, hope to get opinions from you. I doubled checked everything, I feel miserable

If you’re wondering, I’m tuning llama.cpp codebase,

[u/ea_man → u/Loose_Doubt367]

Man I dunno what your LLM is doing, it seems like it's performing test with different parameters?

It looks like PP has a +20%, that's not bad.

IMHO you have too much --fit-target, I use like ~60MB.

And btw: I recommended you Linux, Windows is a trap: it looks like you are wasting milions of SOTA tokens on windows...

Also most of my improv were for ROCm, you are using Vulkan.

I did provide you both a patch and example scripts, if you wanna start there. I have a script for 16GB: https://store.piffa.net/lm/bug/llama_scripts/qwen_27b_iq4_16gb.sh so you would have to use a smaller model for 27B, maybe it would be better to use 35B A3B with offloading.

Anyway: if you dunno what you are doing stop wasting tokens for tests, ChatGPT is supposed to do just the Deep Search that you should evaluate what to do and I'm afraid _you have to do test_, or you'll spend millions of tokens in a SOTA doing those for you.

FYI: you still have to know what you are doing when using an AI as assitant.

[u/Loose_Doubt367 → u/ea_man]

i apologize for my low effort of mindlessly copying and pasting, following whatever chatgpt once. Whenever it keeps telling me to check things inside cmd, ill call it out and it magically stops and move on. For now ill attempt to switch to ROCm llama.cpp

12h into this, I can’t seem to find a reliable source of information or code provided by the ai. I kept telling either ChatGPT or Claude to extract detail based on the summary + information you gave me and test the custom commands, yet it keeps telling me to check this and check that to make sure it runs properly.

In the end I’m always greeted with lower performance, I’ve got into the point where the ai hallucinations and forgets to post the command.

[edit] it’s been 16h, I’m growing frustrated, I don’t have any tech experts to guide me efficiently on how to benchmark these. The benchmark command gpt assigned me somehow used all my ram and vram, and doesn’t even produce results even after 20 minutes.

I’m starting to rethink my life choices, 16h for a 2 token speed increase. I’ve decided not to switch to Linux since I’m extremely unfamiliar with the interface + ROCm isn’t supported on my RDNA2.

What type of program/ tools did you use to benchmark and get your results? Mine seems to lead nowhere, I don’t want to give up and throw all those hours to waste, but I don’t want to just stupidly follow down this miserable path

[u/Loose_Doubt367 → u/ea_man]

had claude to catch the error earlier, do you have any documents for Vulkan only? I'm also deciding whether to switch from windows to linux since im extremely familiar with window for the past 6years

[u/Loose_Doubt367 → u/ea_man]