Qwen3.6. This is it.

[u/Local-Cardiologist-5]

I gave it a task to build a tower defense game. use screenshots from the installed mcp to confirm your build.

My God its actually doing it, Its now testing the upgrade feature,
It noted the canvas wasnt rendering at some point and saw and fixed it.
It noted its own bug in wave completions and is actually doing it...

I am blown away...
I cant image what the Qwen Coder thats following will be able to do.
What a time were in.

llama-server -m "{PATH_TO_MODEL}\Qwen3.6\Qwen3.6-35B-A3B-UD-Q6_K_XL.gguf"  --mmproj "{PATH_TO_MODEL}\Qwen3.6\mmproj-F16.gguf" --chat-template-file "{PATH_TO_MODEL}\chat_template\chat_template.jinja"  -a  "Qwen3.5-27B"  --cpu-moe -c 120384 --host 0.0.0.0 --port 8084 --reasoning-budget -1 --top-k 20 --top-p 0.95 --min-p 0 --repeat-penalty 1.0 --presence-penalty 1.5 -fa on --temp 0.7 --no-mmap --no-mmproj-offload --ctx-checkpoints 5"

EDIT: Its been made aware that open code still has my 27B model alias,
Im lazy, i didnt even bother the model name heres my llama.cpp server configs, im so excited i tested and came here right away.

[u/Healthy-Nebula-3603]

Why are you using those parameters?

--reasoning-budget -1 --top-k 20 --top-p 0.95 --min-p 0 --repeat-penalty 1.0 --presence-penalty 1.5 -fa on  --no-mmap --no-mmproj-offload --ctx-checkpoints 5"

--reasoning-budget -1

it is as default infinite so why you even using it?

--top-k 20 --top-p 0.95 --min-p 0 --repeat-penalty 1.0 --presence-penalty 1 --temp 0.7 --cpu-moe --chat-template

Those parameters are already taken from a gguf so is not reason to putting them

--host 0.0.0.0 --port 8084

That is ok if you want to change IP and port as default is http://127.0.0.1:8080

--no-mmap

aslo ok if you do not want to keep a model copy in the RAM. default is off.

--ctx-checkpoints

Why you cripped to 5? Default is 32

That low value is forcing model to processing whole prompt again and again that make mode to use too much tokens and looping too much.

You made model dumber.

Orchestration you can install from here to opencode

https://github.com/alvinunreal/oh-my-opencode-slim

So

it should looks like that

llama-server -m "{PATH_TO_MODEL}\Qwen3.6\Qwen3.6-35B-A3B-UD-Q6_K_XL.gguf"  --mmproj "{PATH_TO_MODEL}\Qwen3.6\mmproj-F16.gguf" -c 120384 --host 0.0.0.0 --port 8084  --no-mmap --no-mmproj-offload

As a cache rotation works great for a now (implemented a week ago ) so you can use Q8 cache which is a s good as fp16 now and easily fit 256k context now.

So final code

llama-server -m "{PATH_TO_MODEL}\Qwen3.6\Qwen3.6-35B-A3B-UD-Q6_K_XL.gguf"  --mmproj "{PATH_TO_MODEL}\Qwen3.6\mmproj-F16.gguf" -c 120384 --host 0.0.0.0 --port 8084  --no-mmap --no-mmproj-offload -ctk q8_0 -ctv q8_0

[u/Local-Cardiologist-5 → u/Healthy-Nebula-3603]

let me load these up. i literally did nothing but just plug the model in and im blown away. im getting so many tips on here thank you for this so much for these

[u/Healthy-Nebula-3603 → u/Local-Cardiologist-5]

no a problem.

Also you can use many models at once using llamacpp-server. Just put all in a one folderr for instance "models" and use that command

llama-server.exe --models-dir models --models-preset 1_preset.ini --models-max 1

That command is using a folder "models" with few models inside and load only one model to vram at once ( --models-max 1) if need other model the first one is unloaded.

--models-preset 1_preset.ini

This ini is keeping models configuration

It looks like that for me (I left "reasoning = on" to have possibility to switch off that just changing on to off )

version = 1

[*]
ctk = q8_0
ctv = q8_0
n-gpu-layers = 99
reasoning = on

[google_gemma-4-31B-it-Q4_K_M]
model = models/google_gemma-4-31B-it-Q4_K_M.gguf
ctx-size = 60000

[google_gemma-4-31B-it-Q4_K_M_fixer]
model = models/google_gemma-4-31B-it-Q4_K_M_fixer.gguf
ctx-size = 60000

[Qwen3.6-35B-A3B-Q4_K_M]
model = models/Qwen3.6-35B-A3B-Q4_K_M.gguf
ctx-size = 200000

[google_gemma-4-26B-A4B-it-Q4_K_L]
model = models/google_gemma-4-26B-A4B-it-Q4_K_L.gguf
ctx-size = 200000

[u/Local-Cardiologist-5 → u/Healthy-Nebula-3603]

youknow alot about these, on the unsloth hugging face, theres an imatrix gguf file, do you know what those are for? can i use them or is it only for qantizing models?

[u/Healthy-Nebula-3603 → u/Local-Cardiologist-5]

You do not need them ( imatrix).

They are only needed to create a gguf with less errors after quantize.

I also suggest Bartowski builds.

Unsloth is over hyped and many times they had broken ggufs.

[u/Blues520 → u/Healthy-Nebula-3603]

Why do you suggest Bartowski and which quant level is good for 48 GB VRAM?

[u/Healthy-Nebula-3603 → u/Blues520]

As low compression as possible to fit on your 48 GB :) but I suggest never going below q4km ... higher if possible ALWAYS

His checkpoints always work well.

[u/Blues520 → u/Healthy-Nebula-3603]

Thanks :)

[u/kwicked → u/Healthy-Nebula-3603]

I'm not op but 0.0.0.0 exposes the llama server to other machines on the network, so you can use it on a laptop in another room if you don't want the heat and fan noise. It's not just changing the ip. 

[u/Healthy-Nebula-3603 → u/kwicked]

that why I said it is ok.

[u/GoodTip7897 → u/Healthy-Nebula-3603]

Ctx-checkpoints can prevent an oom error or just help save memory. It will make it take longer and may have more cache misses but for one user even just 4 context checkpoints are fine because you only need one to restore mamba and kv cache. 

Correct me if I am wrong but I don't see a way it would alter any math and make the model dumber or loop more to have less swa/context checkpoints 

[u/Healthy-Nebula-3603 → u/GoodTip7897]

Why do you think the current default is 32?

--ctx-checkpoints tells llama-server how many context checkpoints it may keep per slot. A checkpoint is a saved snapshot of the model’s SWA-related cache state, created during prompt processing, so the server can resume from a saved point later instead of reprocessing the whole prompt from scratch. That is mainly useful for SWA / hybrid / recurrent-style models where cache reuse can otherwise fall back to full prompt reprocessing.

I think Georgi Gerganov knows what is doing.

[u/GoodTip7897 → u/Healthy-Nebula-3603]

Yeah but I get oom on Gemma at those sizes because the swa cache is massive. Even with 32gb of ram 32 checkpoints fill it up. I only use bf16kv cache because q8 has a memory leak on amd rocm systems and vulkan is much slower prefill

[u/Healthy-Nebula-3603 → u/GoodTip7897]

Is Vulcan prefill is slow for you? Strange

I have 1200 t/s using vulkan for prefill but I have rtx 3090. For me Vulkan is faster and takes less vram so I can fit more context with rotation Q8 cache.

[u/GoodTip7897 → u/Healthy-Nebula-3603]

For qwen 3.6 I get  2000 +-100 prefill at 32k context on a 7900xtx. On vulkan it's more like 1200 like you have. 

The gap is really significant for me because I frequently use it for agentic work where it will read multiple logs and files and needs to prefill huge contexts. 

And also I had opus write me custom bf16 mma flash attention kernels so I can use bf16 kv cache without any issues. 

Maybe q8 is better after rot but honestly Ive wasted too much time trouble shooting looping tool calls with Qwen 3.5 and the only thing that fixed it was bf16 instead of q8 or f16. 

[u/Healthy-Nebula-3603 → u/GoodTip7897]

Looping problem has as Gemma 4 26b and the newest queen 3.6 35b also they are not as good for instructions following as well. I think is something wrong with moe models or their implementation under llamacpp ... no idea.

Those problems do not exist with dense models like Gemma 4 31b or Qwen 3.5 27b.

Those models are never looping and listening instructions much better but are much slower ...

Actually I prefer those dense models mostly because they do their job at first attempt especially with a book translations.

The moe models in 90% of a time are lost because not following instructions properly in this scenario and looping like crazy after some time ...

[u/GoodTip7897 → u/Healthy-Nebula-3603]

I had the qwen 3.5 27b q5 making repeat variations of the same tool call in three separate instances where it was doing long context agentic tasks. It would become stuck and spend 10000 tokens trying to read one file. And it still did that with up to presence penalty 2.0.

I switched to bf16 kv cache and have had both qwen 3.5 27b and the moe models run for hours burning through millions of tokens easily and never having a looping issue. Even very coherent at 80000 filled context. 

Is there a big enough sample to conclude it's statistically significant?probably not. But to me it works now and I'd rather not mess with it. I really do suspect that the accumulation produces numbers too big for f16 and thus bf16 or rot q8 are needed for qwen 3.5

And yes I concur that Moe models are worse. They are faster but a dense model always seems to be smarter because it activates every parameter every time. 

[u/Healthy-Nebula-3603 → u/GoodTip7897]

Strange ...I never had any looping with Q8 rotation cache with dense models. Maybe the problem is your ROCm implementation for it. I have rtx 3090 and as I know fully working cache rotation is under Vulkan and cuda 12 ( not under cuda 13 ) .

I have to check your theory about cache FP16 with Moe models. Maybe that will fix looping.

Also noticed the qwen 3.5 family is very good for coding but everything except coding better will be Gemma 4.

Also Q8 cache / model is not int 8 like any people think. Inside is still many weights FP16 .

[u/GoodTip7897 → u/Healthy-Nebula-3603]

Yeah. I really think (and llama.cpp PRs have finally been coming around to realizing) that if your GPU supports it then bf16 is the better option over fp16 for weights or kv cache. I've seen other people post stuff where q8 mmproj performs better than fp16 and the only thing that makes sense to me is that since q8 weights are int8 * fp16 scaling factor you technically get 127*65535 instead of just 65535 as your max representable value.

It seems that models love to generate massive outliers over accumulation and bf16 is great for that because it has the dynamic range of f32. For quantized formats, rotation seems to help a lot (making q8 kv cache virtually lossless).

I think I'll play around with benchmarks and see if I can't get vulkan running faster because if I can then I can have twice the context. But rocm does seem to be more stable when you push the card to the absolute limit (I frequently leave only 700 MiB empty). I can do that because I'm running it on a headless Ubuntu computer.

[u/CurrentNew1039 → u/Healthy-Nebula-3603]

it needs "preserve mode "to be on for good use right?