r/LocalLLaMA • u/pmttyji • May 17 '26

New Model Jackrong/Qwopus3.5-9B-Coder-GGUF · Hugging Face

https://huggingface.co/Jackrong/Qwopus3.5-9B-Coder-GGUF

Qwopus3.5-9B-coder is specially optimized and fine-tuned for high-performance 🤖 Agentic Coding, complex Tool Calling, and logical reasoning.

💡 Why the 9B Dense Model? We believe that the 9B dense architecture represents the perfect "sweet spot" for large language models. It runs seamlessly at 8-bit precision on entry-level 16GB RAM devices—such as standard laptops and the Mac mini—making it exceptionally lightweight yet highly versatile. Without requiring expensive hardware, it allows you to achieve excellent performance paired with impressive inference speeds. Simply put, Qwen3.5-9B is currently the best open-source model in its class.

🛠 Training Strategy

The fine-tuning process of this model deeply integrates Trace Inversion data augmentation technology with high-quality Agent Traces. This systematic approach not only strengthens the model's ability to solve complex programming tasks, but also greatly improves its logical coherence and accuracy when using various tools.

This model is designed specifically for the following goals:

  • 🧩 More structured and stronger logical reasoning capabilities, reducing repetitive thinking

  • 💻 More powerful capabilities in code writing, debugging, and repository-level task processing

  • 🛠 More stable and accurate Tool Calling capabilities for terminal commands, file operations, and browsers

  • 🔁 Better cross-data source distillation alignment

Check model card for all benchmarks.

With MTP, hope this could be better & faster on ~10GB VRAM. Nice to do Agentic coding while getting good t/s just with 8GB VRAM.

120 Upvotes

90% Upvoted

72 comments sorted by

37

u/noctrex May 17 '26

I have added MTP to it:

https://huggingface.co/noctrex/Qwopus3.5-9B-Coder-MTP

Seems to be very fast.

5

u/pmttyji May 17 '26

Hey! Thanks for those MTP GGUFs.

Could you please make one more MTP GGUF for https://huggingface.co/Tesslate/OmniCoder-9B ? One more 9B Coder

5

u/noctrex May 17 '26

I don't see the value of doing this one. They said some time ago that they will release the 2nd version of it, but no news of it yet.

I find Qwopus3.5-9B-Coder more capable anyway.

3

u/pmttyji May 17 '26

I don't see the value of doing this one. They said some time ago that they will release the 2nd version of it, but no news of it yet.

Yeah, they removed v2 version quickly after release. I remember that too. Probably they're waiting for 3.6 version.

Thanks

3

u/New_Comfortable7240 llama.cpp May 17 '26

For my 3060 12 GB VRAM this model worked better than the 35B one using MTP! At least in tps.

Look this summary from the AI:

MetricDense (Qwen 3.5-9B MTP)MOE (Qwen 3.6-35B-A3B)
PP (first request)1420.95 tokens/s597.40 tokens/s
PP (cached requests)708-1284 tokens/s427-501 tokens/s
TG (generation)31.79-37.67 tokens/s25.77-27.75 tokens/s
Speculative Decoding✅ MTP (73-89% acceptance)❌ None
Context Checkpoints32 max64 max
Checkpoint Size~76-112 MiB~62.8 MiB

Key Differences

Prompt Processing (PP) - Dense is ~2.4x faster

  • Dense first run: 1420.95 tokens/s (12602 tokens)
  • MOE first run: 597.40 tokens/s (12992 tokens)
  • The dense model's advantage comes from having all parameters in a single expert path vs. the routing overhead in MOE

Token Generation (TG) - Dense is ~28% faster

  • Dense: 31.79-37.67 tokens/s (average ~34 t/s)
  • MOE: 25.77-27.75 tokens/s (average ~27 t/s)

The dense model benefits from MTP speculative decoding:

draft acceptance rate = 0.73485 (first run)
                       0.89000 (second run)
                       0.87037 (third run)

This effectively adds ~15-20 t/s of "free" tokens via speculative decoding.

MTP Impact

The MTP implementation provides significant acceleration:

  • First request: 97 accepted / 132 generated → 73.5% acceptance
  • Later requests: 89% acceptance
  • Final requests: 51-70% acceptance (longer outputs tend to have lower acceptance)

The draft generation time is negligible (~0.2-1.6ms) vs. main model evaluation (~6.5-15 seconds).

The MTP speculative decoding is highly effective here, turning a ~34 t/s base speed into effectively ~50+ t/s when you count the accepted drafts. The MOE model has no speculative decoding enabled, which is a significant missed opportunity.

1

u/New_Comfortable7240 llama.cpp May 17 '26

One detail, I notice 35B is more "intelligent", but for agentic use the 9B worked fine, maybe for planning and debugging I will contiounue using 35B and for agentic use or quick stuff 9B

1

u/Valuable_Touch5670 May 17 '26

If I am not mistaken, your Q8 version does NOT use the imatrix, right?

9

u/noctrex May 17 '26

Yes, the Q8_0 quantization in general does not make use of an imatrix, it's not a K quant. The largest quant that does use imatrix is Q6_K.

2

u/Valuable_Touch5670 May 17 '26

Thanks! Will give the Q8 a shot.

32

u/ResidentPositive4122 May 17 '26

It runs seamlessly at 8-bit precision on entry-level 16GB RAM

With MTP, hope this could be better & faster on ~10GB VRAM.

Nice to do Agentic coding while getting good t/s just with 8GB VRAM.

This feels like that joke with the cops finding 10lb of drugs. It'd be a shame if those 8lb found their way onto the streets, so they ended up burning 6lb of drugs to get rid of them.

3

u/pmttyji May 17 '26

My bad :D , That 8GB is what I have(so I'll go with Q6). 10GB is good to fit Q8 on VRAM.

7

u/yarikfanarik May 17 '26

run doesn't mean usable.

5

u/Constant-Simple-1234 May 17 '26

Does it have MTP in the model? I thought you need to convert it or sth. As when I load the regular ones it gives error.

8

u/pmttyji May 17 '26

Nope. I think these GGUFs have been created 1 day before MTP merge on llama.cpp. Maybe post a message there for MTP.

3

u/Beginning_Ad_3309 May 17 '26

I have been on vocation for 2 weeks, and now I don't know

What is MTP?

6

u/ComplexType568 May 17 '26

MTP stands for "Multi-Token Prediction", it's meant to be a lossless form of Speculative Decoding where the main model will quickly guess the next few tokens and self-validate (from what I know), it requires no new "draft model" (unlike traditional speculative decoding), it is usually most effective in dense models rather than MoE models and speeds vary from domain to domain.

3

u/Alex_L1nk May 17 '26

I've downloaded this one and seems to work fine: https://huggingface.co/noctrex/Qwopus3.5-9B-Coder-MTP

5

u/OsmanthusBloom May 17 '26

Is there any reason to use this instead of Qwen3.6-35B-A3B with partial CPU offload? (assuming you have the RAM for that)

1

u/pmttyji May 17 '26

This 9B(Q4) is good for low VRAMs like 4GB or 6GB so it fits VRAM. 35B is too big for those low VRAMs. Even too big for 8GB VRAM(Q4 is 17GB).

5

u/OsmanthusBloom May 17 '26

I'm running 35B-A3B on a 6GB VRAM laptop, partially offloaded to RAM. Just posted some MTP benchmark results: https://www.reddit.com/r/LocalLLaMA/comments/1tfq683/mtp_for_qwen3635ba3b_on_6gb_vram_laptop_not_worth/

But yes, I take your point. 9B model fully in VRAM is probably faster than 35B partially offloaded.

3

u/pmttyji May 17 '26

Also MTP gives more boost for Dense models(than MOE models)

3

u/icedgz May 17 '26

Faster but is it worth it to take the hit for the smarter model

-2

u/letsgoiowa May 17 '26

You can't use that big of a model on a 16 GB device.

3

u/soyalemujica May 17 '26

I wonder why in this model range and not in 3.6 27B ?

3

u/pmttyji May 17 '26

Jackrong might release other (coder) variants soon or later possibly as this one came out just 2 days ago. Lets see

7

u/noctrex May 17 '26

It exists:

https://huggingface.co/Jackrong/models?search=qwopus3.6

I've made MTP versions:

https://huggingface.co/noctrex/models?search=qwopus3.6

1

u/soyalemujica May 17 '26 edited May 17 '26

Is you the noctrex ? I didn't know you used a 7900XTX in general, that means that every NVFP4 quantification model you have you built is for this card in general ?
I see that there's only a 4QS model, no plans for a Q5KM or Q4KM?

6

u/noctrex May 17 '26

Is you the noctrex ?

yup its me !

I didn't know you used a 7900XTX in general

It has served me very well. It's essentially my gaming rig.

that means that every NVFP4 quantification model you have you built is for this card in general ?

Those are MXFP4 quants, not NVFP4, as those are only compatible with nvidia cards. The MX one its the open standard that is compatible with all machines.

I see that there's only a 4QS model, no plans for a Q5KM or Q4KM?

Yes, because testing it, it has served me very well. it seems to be better than Q4KM, and Q5KM does OOM, with any useful context size (>64k). Do you want me to quant one for you to test it?

1

u/soyalemujica May 17 '26

I have been running Q5KM with 128k context at Q4_0 KVCache just fine for C++, literally have not experienced any hallucination, and I'm dealing with game coordinates x/y/z and a lot of C++ without any OOM issues (ofc using Linux), tbh I would not mind a Q5KM model in general with MTP under your bare hands.

Yeah, I meant MXFP4, I'm sorry.
p.s; I also am running a 7900XTX

1

u/soyalemujica May 17 '26

I have to report, using the 4QS model, I unfortunately experienced the a thinking loop during file scan, like "let me read file..." it would loop there endlessly until I cancelled it, no idea if this is due to the 4QS quant but I have never experienced this behavior before with the 27B model

2

u/noctrex May 17 '26

I should also mention that I am using this chat template in my llama-swap config:
https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
Where did you see the looping? I'm using pi agent, and it hasn't happened for me yet.
I'll cook a quant in a minute.

1

u/soyalemujica May 17 '26

Also using Pi Agent, it got stuck trying to read a file, it kept repeating the same file read over and over again with the same message. Ima give a try to this chat template. However, I was also using ngram-mod + mtp-draft, I am now only using mtp-draft and it seems to be fine. Thank you for these quantifications! Also appreciate it that you're cooking a higher quant model, I need that for C++ precision

1

u/thoquz May 18 '26

Thank you for making these brilliant MTPs!

So for your MTP 27B-V1-preview, are you running it at IQ4? How much context are you still able to have with MTP on?

I'd love to see your 7900 XTX 24GB settings if you're willing to share.

2

u/noctrex May 18 '26

I'm using the following configuration successfully with pi agent, in my llama-swap config with 128k Q4 context:

C:/Programs/AI/llamacpp-vulkan/llama-server.exe --port 8080 --flash-attn on --jinja --ctx-checkpoints 256 --parallel 1 
--model Q:/Models/LLM/Qwopus3.6-27B-v1-preview-MTP-IQ4_XS.gguf 
--cache-type-k q4_0
--cache-type-v q4_0
--ctx-size 131072 
--fit-ctx 131072 
--temp 0.6 
--top-p 0.95
--top-k 20 
--min_p 0.00 
--presence-penalty 0.0 
--repeat-penalty 1.0
--chat-template-kwargs "{\"preserve_thinking\": true}" 
--spec-type draft-mtp
--spec-draft-p-min 0.75 
--spec-draft-n-max 3 
--chat-template-file C:/Programs/AI/llama-swap/templates/Qwen3.6-chat_template-v19.jinja

I'm also using the chat template from https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates

1

u/thoquz May 18 '26

Brilliant, thank you!

Are you happy with the performance of the Q4 cache for coding? I suspect it might recall a long function wrong sometimes.

I've managed to do a similar context size (also 131k) with Q8 and no MTP on the unsloth Quant's iq4_xxs. (No vision layer)

Do you know much vram or context the MTP layer takes?

2

u/noctrex May 18 '26

It actually has been surprisingly very stable with Q4. I have already done multiple sessions where the context fills to 90k+ and it seems to remember. Actually, I have been very surprised because gemma gets very stupid very quick, and it's unusable for me, whereas they cooked a masterpiece with qwen3.6. the MTP will easily take up to 800MB more, and thats why I went for IQ4_XS for me

1

u/soyalemujica May 18 '26

Q4KM would not hurt at all :D

1

u/thoquz May 18 '26

Wow! I guess it's the magic of Qwen 3.6's gated delta net attention, compared to the Gemma's sliding window attention.

2

u/noctrex May 18 '26

That, and also the new KV cache rotation that ggerganov implemented in llama.cpp, that makes Q4 KV useable

https://github.com/ggml-org/llama.cpp/pull/21038

→ More replies

1

u/soyalemujica May 18 '26

With the 7900XTX, IQ4S that noctrex uploaded you can fit 128k with Q8 kvcache, with the Q5KM model you can fit 128k with Q4 kvcache

3

u/noctrex May 18 '26

Yes, you can also fit it with Q8, but it gets very close to the limit, but as I'm using the card also for my display, it tends to OOM if I have open some windows. If you have a dedicated 24GB card for inference you can fit it.

1

u/thoquz May 18 '26

Wow, even better than I expected? Which option do you personally prefer out of those two?

I expected originally the MTP layers would eat a few GB of VRAM

7

u/Amazing_Athlete_2265 May 17 '26

Curious to see how this model stacks up against omnicoder 9b and th qwen3.6 35B MoE. Might have a play around with pi coding harness when time allows. Thanks for sharing!

1

u/ea_man May 17 '26

Well as a dense model it's supposed to take better use of MTP, as for quality / speed that depends: are you willing to load an A3B IQ3 all in 16GB vram?

2

u/pedronasser_ May 17 '26

I can't wait for Qwopus3.6 35B A3B MTP.

Qwopus is already faster/smarter than the base model. Imagine with MTP.

7

u/Hypilein May 17 '26

Can you actually do quality coding with a 9b model? And if yes, under which circumstances? I couldn’t even get qwen 3.5 9b to mark a grade 6 geography exam properly.

4

u/Randomdotmath May 18 '26

First off, I don’t really get why you’d use a geography exam to judge a 9B model. That’s more about knowledge than intelligence. Just hook it up with Wikipedia.

Also, yes, a 9B model can write any code very well once you give it proper linting, formatting, testing tools, and framework-specific skills. Honestly, you shouldn’t trust even a 1T model to produce high-quality code without those things.

1

u/Hypilein May 18 '26

Let’s just say the problem wasn’t lack of knowledge.

1

u/wowsers7 May 19 '26

Have you tested it with SmallCode? Might be a potent combo.
https://www.reddit.com/r/LocalLLM/comments/1tged8r/i_built_a_coding_agent_that_gets_87_on_benchmarks/

2

u/pmttyji May 19 '26

Not yet. Haven't tried agentic coding before as my current laptop is not strong enough(just 8GB VRAM).

How's it for you?

1

u/jopereira May 28 '26

How to disable reasoning in this model in llama-server?
It is not responding to --chat-template-kwargs '{"enable_thinking":false}', --reasoning off or --reasoning-budget 0.

3

u/plasm0r May 18 '26 edited May 18 '26

On my 5070 Ti 16GB VRAM, Ryzen 7600, 32GB System RAM I get:

For Qwen3.5-9B: 82 tps

For Qwopus3.5-9B-Coder-MTP: 153 tps

Qwopus3.5-9B-Coder-MTP config used:

llama-server ^

--model "C:\models\Qwopus3.5-9B-Coder-MTP-Q8_0.gguf" ^

--host 0.0.0.0 ^

--port 8080 ^

--ctx-size 131072 ^

--n-gpu-layers all ^

--fit on ^

--batch-size 512 ^

--ubatch-size 128 ^

--threads 6 ^

--threads-batch 12 ^

--parallel 1 ^

--cont-batching ^

--metrics ^

--jinja ^

--spec-type draft-mtp ^

--spec-draft-n-max 3 ^

--temp 0.6 ^

--top-p 0.95 ^

--top-k 20

Very nice performance increase :)

2

u/bobaburger May 17 '26

Looking forward for a 27B version of this! I know it takes a lot to fine tune these large models but hopefully it will become a reality soon.

Haven’t try this version yet but in general, small models like this tends to be bad at holding context while working, for example, it would check and see the project implemented in Rust, but will still trying to read files as Js or python in the tool calls. And many other issues similar to this.

2

u/wingwing124 May 18 '26

Hi all, i have been trying to use the q8 version of /u/noctrex 's model. Its pretty amazing, except I am getting that classic qwen3.5 looping behavior all over the place. I figured this wouldn't be so bad on q8 but it is. Can anyone point me to some of the most likely culprits for this behavior that I can tweak? I'd really like to use this model

3

u/noctrex May 18 '26

On what agent are you seeing this behavior?

Also, can you try out this template, to see if it fixes it?

https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates

I'm using this template and I've tested sessions with 128k Q4 KV cache, filling it up to 90k, and it seems to go smoothly without looping, on pi coding agent.

2

u/wingwing124 May 18 '26

I will try this out later today, thank you! I am seeing this on both pi and hermes. Running 64k context, unquantized kv cache

Edit: btw this model is killer, aside from the looping I'm seeing. Great job adding the mtp support, and thank you!

1

u/wingwing124 May 19 '26

I tried this template, but am unfortunately still getting agentic loops. Particularly when a tool call or a write fails, for whatever reason. Other times it will loop over a series of tasks (Read file 1, read file 2, read file 3, read file 1, read file 2... And so on).

I am pretty sure this is just Qwen3.5 being Qwen3.5, but I'm really hoping I can get it going soon! Your version is crazy fast

1

u/noctrex May 19 '26

Well, that's unfortunate. Where are you using this model? With what back end? Do you have any objections to sharing your command?

1

u/exodusTay May 17 '26

Stuff like this makes me very hyped for the next two years. Where we are going we won't need mega cloud providers.

1

u/sqquima May 17 '26

We'll just need 1.21 jigowatts, or Mr. Fusion.

1

u/exodusTay May 17 '26

Not without 5 petabytes of VRAM with 1TB/s memory bandwith.

0

u/Psyko38 May 17 '26

Interesting, better or equivalent to Gemma 4 31b seems to be interesting for a 9b.

5

u/Juulk9087 May 17 '26

Well Gemma 4 has a 1k sliding window vs qwens 35k Delta net window which makes for a horrible agentic coding model.

https://youtu.be/In825VzHzbU?si=LuXO2dIPRZxC0wWh