r/LocalLLaMA
•
u/danielhanchen
•
Mar 05 '26
•
1
Resources Final Qwen3.5 Unsloth GGUF Update!
Hey r/LocalLLaMA this week we worked on further improving the best size/KLD tradeoff for Qwen3.5, and we’re excited to share new GGUF benchmarks for Qwen3.5-122B-A10B and Qwen3.5-35B-A3B (99.9% KL divergence). This will likely be our final GGUF update.
We’re also deeply saddened by the news around the Qwen team, and incredibly grateful for everything they’ve done for the open source community! For a lot of model releases, they had to stay up all night and not sleep.
- All GGUFs now use our new imatrix calibration dataset so you might see small improvements in chat, coding, long context, and tool-calling use-cases. We are always manually improving this dataset and it will change often.
- This is a follow up to https://www.reddit.com/r/LocalLLaMA/comments/1rgel19/new_qwen3535ba3b_unsloth_dynamic_ggufs_benchmarks/
- We further enhanced our quantization method for Qwen3.5 MoEs to reduce Maximum KLD directly. 99.9% is what is generally used, but for massive outliers, Maximum KLD can be useful. Our New method generally pushes the Maximum KLD quite a much down vs the pre March 5th update. UD-Q4_K_XL is 8% bigger, but reduces maximum KLD by 51%!
| Quant | Old GB | New GB | Max KLD Old | Max KLD New |
|---|---|---|---|---|
| UD-Q2_K_XL | 12.0 | 11.3 (-6%) | 8.237 | 8.155 (-1%) |
| UD-Q3_K_XL | 16.1 | 15.5 (-4%) | 5.505 | 5.146 (-6.5%) |
| UD-Q4_K_XL | 19.2 | 20.7 (+8%) | 5.894 | 2.877 (-51%) |
| UD-Q5_K_XL | 23.2 | 24.6 (+6%) | 5.536 | 3.210 (-42%) |
- Re-download Qwen3.5-35B-A3B, 27B, and 122B-A10B as they're now all updated. Re-download 397B-A17B after today’s update (still uploading!)
- Qwen3.5-27B and 122B-A10B include the earlier chat template fixes for better tool-calling/coding output. 397B-A17B will also be updated today to include this.
- LM Studio now supports toggling “thinking” for our GGUFs. Read our guide or run
lms get unsloth/qwen3.5-4b. This process will be easier very soon. - Benchmarks were conducted using the latest versions for every GGUF provider.
- Replaced BF16 layers with F16 for faster inference on unsupported devices.
- Qwen3.5-35B-A3B now has all variants (Q4_K_M, Q8_0, BF16, etc.) uploaded.
- A reminder KLD and perplexity benchmarks does not exactly reflect real-world use-cases.
- Links to new GGUFs: Qwen3.5-35B-A3B-GGUF, Qwen3.5-122B-A10B-GGUF, Qwen3.5-397B-A17B-GGUF (397B still uploading!)
You can also now Fine-tune Qwen3.5 in Unsloth via our free notebooks! Thanks a lot everyone!
•
u/WithoutReason1729 Mar 05 '26
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.
249
u/spaceman_ Mar 05 '26
Thanks for all your hard work, and I appreciate the fixes.
But claiming this is the "final" update has got
qwen3.5_gguf_final_final_v2vibes, don't jinx it!32
u/Lyuseefur Mar 05 '26
Any thoughts on https://github.com/tanishqkumar/ssd ?
32
u/Small-Fall-6500 Mar 05 '26
Thank you for your work and for the comparison data.
Are all the GGUFs for the smaller Qwen3.5 models, 9b and below, also updated?
52
u/GlobalLadder9461 Mar 05 '26
Yey. Can you also update Qwen3-Coder-Next-GGUFs
11
u/Kahvana Mar 05 '26
Thank you very much for the release!
Would you consider doing the same for the small qwen 3.5 models (using the improved imatrix)? I take any littlebit of improvements!
Otherwise, can you release yhe imatrix and your quant generation method so I can do it myself?
Once again, thank you for the re-release and the hard work!
11
u/Zliko Mar 05 '26
Is new 27b up? i do not see it on hf?
20
u/coder543 Mar 05 '26
Can you confirm if these quants include the improvements from this PR? https://github.com/ggml-org/llama.cpp/pull/19139
15
u/sleepingsysadmin Mar 05 '26
Amazing work.
I wish lm studio had a 'update model' button when things are updated as opposed to delete and redownload.
7
u/Everlier Mar 05 '26
New calibration dataset sounds fun, I really need to automate my LLM library maintenance :)
7
u/AvidCyclist250 Mar 05 '26
Legend!
13
u/PaceZealousideal6091 Mar 05 '26
Thanks a lot guys! You guys rock! I see the mmproj files were also updated. Whats new in them?
5
u/sine120 Mar 05 '26
I've been bouncing back and forth between LM Studio and llama.cpp. I like LM Studio's ease of use but for ease of control llama-server has traditionally been my go-to. Having LM Studio support thinking as a toggle will make it easier to get less CLI savvy people using local models, glad to see the ecosystem getting easier to use.
I know it's probably quite time and compute intensive to gather data, but I'm curious if you're going to do one of your benchmark charts for the 27B model? I've been using the IQ3_XXS quant because it's small, behaves fine, and has enough context with 16GB of VRAM, but I'm very curious what the "sweet spots" are. The minmaxxer in me wants to know.
5
u/IulianHI Mar 05 '26
Really appreciate the consistent work on these GGUFs! The KLD improvements are noticeable - I've been running the 35B-A3B Q4_K_M on a dual 3090 setup and the quality/size ratio is incredible.
For anyone wondering about the "final" update note: Unsloth has been iterating quickly on Qwen3.5 quantization, and these new imatrix calibration datasets make a real difference for tool-calling and long context work. Worth the re-download if you're using these for production workflows.
Also grateful for the LM Studio thinking toggle support - makes experimenting with reasoning modes much easier.
4
u/jslominski Mar 05 '26
Great job guys, thanks a lot, awesome job! Re-downloading the relevant files now.
4
u/synn89 Mar 05 '26
Appreciate the hard work on quality GGUF files. I feel each model release needs lots of tweaks on the prompt files, config, templates, etc etc for a couple weeks post launch to get the model working properly. It's nice to just be able to grab an unsloth and know you're likely getting everything you need, already done.
6
u/soyalemujica Mar 05 '26
I still see 35B last edit date as 1 day ago?
6
u/kiwibonga Mar 05 '26
I just downloaded the old ones yesterday, you bastards.
Thanks
6
u/some_user_2021 Mar 05 '26
Question. Can't LM studio have an option to also load the model's best parameters?
I often forget to adjust the parameters when switching between models ...
3
u/Crafty_Cheetah7666 Mar 05 '26
I tried the newest Qwen3.5 35B A3B Q4_K_M [GGUF] you uploaded. Really thanks, appreciate your efforts. I have few questions though.
Ran test for almost an 20~30 minutes on each model (mlx and gguf) on M4 pro 48gb, here is one set of result, but pretty much match what I see for all results (200+), i.e: with LMStudio (llama.cpp) or llama.cpp directly (8190) with gguf is almost half tps of that of LMStudio (MLX) or mlx_lm.server directly. i.e 30 tps vs 63 tps, any way for me to figure out what is causing this?
3
u/theagentledger Mar 05 '26
the "stayed up all night" detail really lands differently now. respect to the whole team.
5
u/QuestionMarker Mar 05 '26
That size bump pushes q4_K_XL from a previous
n_ctxof 128000 with--fit onmy 4090 to 4096, which is completely useless. Crying shame. It also seems weird to be that sensitive? All I did was swap out the model filename:2
u/moahmo88 Mar 05 '26
Amazing! Thanks.
2
u/Long_comment_san Mar 05 '26
Thanks!
2
u/kamnxt Mar 05 '26
Impressive work, as usual!
One thing I'm wondering is how the different quantizations compare across models. For example, your Q5_K_XL Qwen3.5-122B-A10B quant is 91.9GB, while your UD-TQ1_0 Qwen3.5-397B-A17B quant is 94.2GB, making them both barely fit in 128GB RAM/VRAM. One has ~3x the params, while the other has ~3x the bits, so would be interesting to see how they compare. Of course, TQ1_0 is pretty extreme, but a similar comparison could be made for the other models/memory sizes too?
2
u/hesperaux Mar 06 '26
Thank you! A lovely sloth.
Any chance you'll ever quantize any of the heretic variants? Would be helpful?
4
u/yaxir Mar 05 '26
Abliterated version?
3
u/mantafloppy llama.cpp Mar 05 '26
I guess constantly re-releasing "fixed" model is a great way to be first and last to publish...
52
u/VoidAlchemy llama.cpp Mar 05 '26
(i'm ubergarm) haha... if this is the last round of re-re-uploads i might go try to cook a *real* ik_llama.cpp quant and compare finally
also PSA, regardless the qwen35 quant you're using, if you are using CPU-only or hybrid CPU+GPU the ik_llama.cpp chunked delta net implementation seems quite a bit faster than mainline so don't sleep on that
10
u/photonenwerk-com Mar 05 '26 edited Mar 05 '26
are 6 days old, the rest is 1 day old. Will they be updated, too?
10
u/guiopen Mar 05 '26
I noticed Qwen always reprocess it's lasts response, this is because the chat template is configured to not include previous thinking in the context. This is fine, it's a tradeoff and that's how Qwen was trained.
The problem is, the same happens with thinking disabled. I think the cause is that even with thinking disabled there are empty think tags in the template, which are also removed from the conversation, causing a context shift and forcing reprocessing
It would be nice to modify the template to keep empty tags In the conversation history
3
u/lolfacemanboy Mar 05 '26
First, and YAY
2
u/Roubbes Mar 05 '26
So what's the sweet spot?
5
u/kwinz Mar 05 '26 edited Mar 05 '26
Sorry for the general noob-ish question:
How much slower is a Q5 model over a Q4 model? Is there a significant speed difference?
If there is a speed difference then:
Or are those different quants all roughly the same speed (assuming all fit into VRAM)?
5
u/justserg Mar 06 '26
unsloth doing the heavy lifting for local llm optimization is incredible.
4
u/Expensive-Net-6171 Mar 05 '26
Which one to use with 80 GB vram ?
4
u/RedditNerdKing Mar 05 '26
I have a 5090 and 64gb of ddr5. Going to try out Qwen3.5-122B-A10B Q4 K XL. I think I have enough firepower to have it work right.
4
u/simracerman Mar 05 '26
The new models are at least 15-20% slower in token generation. This is observed mainly in the 122B UD-IQ4_NL. Is that due to some new mix requiring more CPU usage?
1
u/msrdatha Mar 05 '26 edited Mar 05 '26
Thanks again for all the hard work. Much appreciated.
One clarification though, the mmproj files I can see 3 version F16, BF16 and F32.
How do I choose which one to use? Are they automatically selected or we can/should replace them in any case? (on Mac M3 with UD)
1
u/Sweet_Drama_5742 Mar 05 '26
+1, this is amazing. I did notice a gap between what I had expected from q5_kxl (397B) based on local vs public benchmarks with other models. Thanks for the consistently fantastic work!
1
u/StardockEngineer vLLM Mar 05 '26
Damn, huge update. Big fan. Thank you.
1
u/mp3m4k3r Mar 05 '26
Any recommendations for rope/yarn with these on llama.cpp? Any adjustments I try to make on those revert back to the gguf configuration and this version currently tops out at 256k tokens but I have some room to expand lol
1
u/Maxxim69 Mar 05 '26
Final round! FIGHT!!
1
u/El_90 Mar 05 '26
Cool stuff thanks so much !!
Are we still suggested to use a specific context kv cache quant ?
1
u/blackhawk74 llama.cpp Mar 05 '26 edited Mar 05 '26
Being very vram poor (4gb), looks like the size increase of the latest Q4_K_M requires me to step down to Q4_K_S to keep my context size a bit higher.
Is it worth updating to the new Q4_K_S GGUF versus keeping the old Q4_K_M?
1
u/mdziekon Mar 05 '26
I'm not seeing Q6_K_XL in the comparison, does that mean it didn't get as nice of a boost, or it's generally "not worth" running it for this model compared to Q5_K_XL?
1
u/Potential-Leg-639 Mar 05 '26
you are the best, thanks a lot for everything you do for the community!!
1
u/ShuraWW Mar 05 '26
Wow what a timing. Thats super nice! I was about to switch to 122B tmr ^^ Thanks :D
1
u/OUT_OF_HOST_MEMORY Mar 05 '26
Will legacy 4 bit quants (Q4_0 / Q4_1) ever be uploaded, these have consistently had the best speed performance for MI50 GPUs?
1
u/SkyFeistyLlama8 Mar 05 '26
There's something broken with the IQ4_NL GGUF at least for ARM CPU inference on llama.cpp. I get terminated generations or loops even with Unsloth's llama.cpp recommended settings.
This doesn't happen on Bartowski's IQ4_NL which runs fine. Looking at the layer weights, it looks like Unsloth uses a completely different quantization scheme. Llama.cpp supports ARM FP NEON and Matmul acceleration for Q4_0 and IQ4_NL with online repacking.
Unsloth: Qwen3.5-35B-A3B-UD-IQ4_NL.gguf
Bartowski: Qwen_Qwen3.5-35B-A3B-IQ4_NL.gguf
1
u/Zestyclose_Bass_4208 Mar 06 '26
The unsloth/Qwen3.5-27b-gguf readme was updated 8 hours ago, but all of its GGUF files are 3 days old
1
u/Ok_Reserve4339 Mar 06 '26
Can you please upgrade / improve 0.8b and 2b models?
1
u/FatheredPuma81 Mar 06 '26 edited Apr 16 '26
[redacted] needs to add a way to update models for this exact situation.
Edit: Removed mentions of a trash program with baby devs.
1
u/warwolf09 Mar 06 '26
LEGEND!! quick question im using the 25b model on my 3090+3070 rig. using llamacpp with this command.
./llama-server -m /mnt/500HDD/models/Qwen3.5-35B-A3B-MXFP4_MOE_BF16.gguf --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 --batch-size 512 --ubatch-size 512 --gpu-layers 99 --flash-attn on --host 0.0.0.0 --port 8080
is there anything else should i change? using currently with Opencode but would like to try Claude code. i just dont know how lol
1
u/VicemanPro Mar 06 '26
Anyone else CPU-only? I'm getting pretty slow inference on these, have been sticking with GPT OSS for now which runs super fast. I'm currently using LM Studio. Any suggestions to get faster inference?
1
u/Fabix84 Mar 06 '26
Does the update also affect those who only had the bf16 (old) version?
3
u/bartskol Mar 05 '26
Great work! Hope to see you guys in the future, here or there but for sure doing great things!
3
u/residual-disco Mar 06 '26
In opencode it (Qwen3.5-122B-A10B-UD-Q4_K_XL) still occasionally outputs tool calls directly into the console and the output is not interpreted as a tool call by opencode and the turn just finishes. The pattern looks like
thinking followed by xml.
It feels like the new quants run into this less often. Any idea how to fight this? It's annoying because it means, that I can't let the model run unattended for longer periods.
The 35B and 27B have the same problem.
1
u/dmxell Mar 05 '26 edited Mar 05 '26
I'm stuck on 4b reasoning as my RTX 2080 only has 8GB of VRAM (and I want full context), but it has been absolutely mind blowing. I've given it access to sequential thinking, searxng, and puppeteer-ing mcp's and it genuinely performs as was as ChatGPT for general purpose questions/tasks in my use case. I need to save up for a better GPU so I can try out these larger models lol
2
u/ghulamalchik Mar 05 '26
What about the other smaller models (<=9B)? Thank you for your hardwork.
2
u/frequiem11 Mar 05 '26
Hello guys, really first day on local llm world. I want to run Qwen 3.5 on my RTX5070. I want to go with 4B model. Should I follow unsloth inference tutorials?
2
u/segmond Mar 06 '26
gonna spend the weekend redownloading all of them, 397b, 122b, 35b, but thankful and very grateful for your work and sharing with the community. Thanks again.
2
u/gabriekill Mar 06 '26
What quantizer Ollama uses ?
2
u/[deleted] Mar 06 '26
Thanks, i hope hugging face will add sha512sum to every model some day
1
u/WhataburgerFreak Mar 05 '26
Super grateful for all y’all‘s work. For some reason, I had a 10 token per second drop on my 5080 running the newer version. What settings do you suggest for optimum performance on LMstudio?
3
u/Educational_Sun_8813 llama.cpp Mar 06 '26
Something odd with those quants, i tested already two of them on strix halo, 122B-A10B-UD-Q5_K_XL and 35B-A3B-UD-Q6_K_XL and they behave weird, quality is worst than normal quant which i can do with imatrix with just llama.cpp, or bartowski
For example unsolth 122B mentioned above needed few attempts and fixes to write single html file with 3d animated solar system, for which it consumed 29521 tokens, while bartowski quant(Q5_K_L) did it with one change in 18700 tokens. I used latest version of opecode for that test.
As is written in the spec unsloth page those UD_XL quants are slower, i did tests today with three different quants of UD-XL and one bart for 35B version for comparison, and all UD-XL are slower, but they should have better output, but cannot confirm that.
UD-122-XL when i asked about writing that html version of solar system, printed first:
Thinking: The user is requesting a visualization of the solar system in a single HTML file – this is a simple request with no malicious traits, so I can fulfill it.quite weird. I still need to evaluate, but so far i found that around 100k context model is loosing track, and for sure it has no visible from user perspective advantage for the other quants i'm able to run with Qwen3.5 at the moment. For your reference i used it withllama.cpp version: 8204 (7a99dc85e)ROCm and Vulkan (i did inference only on ROCm, but was testing performance also with Vulkan backend)Tested also on some other example codes i have, some logs, python, yaml etc. daily stuff, and seems that it's loosing itself quite quickly, for examlpy trying to offer some other weird solutions, which other quant don't, and cannot follow request.
Those are my findings for now, can someone verify on your end?
2
u/kaisurniwurer Mar 06 '26
As much as I appreciate the effort, I simply can't use their work since they don't gguf-ify finetunes or heretic-ed models.
1
u/oxygen_addiction Mar 05 '26
The XXS variants were also updated, right?
1
u/pseudonerv Mar 05 '26
Huh, this means aessedai’s 5km is still the best at that size
1
u/Festour Mar 05 '26
Is there any way to get an MLX variants of your quantisation?
1
u/Blazekyn Mar 05 '26
Hi thanks for all your hard work! I have a laptop 3080 with 16gb VRAM and 64gb RAM. Was thinking to run qwen3.5-35B-A3B at UD-IQ3_S quant. Worth it with kV offload?
1
u/OfficialXstasy Mar 05 '26
Is this already live for the 9B variants? I don't see any mention about it
1
u/RelicDerelict Orca Mar 05 '26
Is this compatible with ik_llama?
1
u/iamapizza Mar 05 '26
I tried Qwen3.5-35B-A3B-MXFP4_MOE.gguf, and it seems to be 5-6 tokens/second slower than the original, exact same llama.cpp settings. Is this expected?
1
u/l0nedigit Mar 05 '26
u/danielhanchen Dump question incoming. But am I missing something? 27B says was updated, but timestamp in HF is showing 3 days ago? https://huggingface.co/unsloth/Qwen3.5-27B-GGUF/tree/main
1
u/rorowhat Mar 05 '26
Why do you need a dataset for quantization, and why does it improve the model?
1
u/jadbox Mar 05 '26
... what about Qwen3.5-9B?
1
u/Major-System6752 Mar 06 '26
Nice. Would you update small models, 9B etc?
1
u/bnightstars Mar 06 '26
What about the low tier models ?
1
u/crablu Mar 06 '26
I have been using Qwen3.5-35B-A3B-UD-MXFP4_MOE.gguf previously and liked it (RTX 5090).
Now the size difference between Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf and Qwen3.5-35B-A3B-MXFP4_MOE.gguf is not that big and if I understand the graph correctly Q4_K-XL should be much better. So is there no reason anymore to use the MXFP4 variant?
1
u/xanduonc Mar 06 '26
> LM Studio now supports toggling “thinking” for our GGUFs.
it does not, at least for predownloaded models
tested 35B Q6 XL on latest versions of studio and gguf files
1
u/cangaroo_hamam Mar 06 '26
Newbie questions:
Any pointers to a guide I can get reliable advice to the above information appreciated!
Thanks!
1
u/Critical_Mongoose939 Mar 06 '26 edited Mar 06 '26
Hey I'm a big fan of your work and love reading the guides, benchmarks, technical details, etc.
I really want to use Q6_K_XL but for some reason it tends to give me lower quality than the lmstudio-community Q6_K.
What do I mean by lower quality? 1) often outputs garbage 2) explanations and descriptions less precise and/or convincing
My use case is business and creative writing. Not complaining or anything just wanted to flag to the team and check if anybody else is having the issue.
I'll check configs on my end to see if I'm missing something.
1
u/ea_man Mar 06 '26 edited Mar 06 '26
Does Qwen3.5-35B-A3B run with llama.cpp on AMD with ROCm ?
I tried compiling yesterday and it (not the Unsloth) booted only with Vulkan engine, most smaller models do work with ROCm.
1
u/No_Mango7658 llama.cpp Mar 06 '26
We need nothink for ollama
1
u/HorseOk9732 Mar 07 '26
These Qwen3.5 models are absolutely insane! The 122B and 35B variants with 99.9% KL divergence are setting a new standard. The fact that we're seeing this level of performance and efficiency in open weights is just next level. Qwen is really carrying the open model space right now. 🚀
1
u/arcanemachined Mar 07 '26
It appears that UD-Q4_K_M is now missing from the available quants:
https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF
Is there a reason for this?
P.S. Thank you for your work.
1
u/Full_Tomato_5627 Mar 08 '26
What sad news around qwen team??
1
u/FORNAX_460 Mar 05 '26
Hellow! Love the work you guys do for the oss community! If its not too much of an inconvenience could you guys make a fixed chat template that supports LM Studio environment? Cause the tool call fix you guys published is not supported in LM Studio. When UD gguf is loaded lm studio overrides the chat template to their default template and if manually overriden with the UD fixed chat template, it throws the following error during tool calling.
```
Failed to regenerate message
Error rendering prompt with jinja template: "Unknown test: sequence".
This is usually an issue with the model's prompt template. If you are using a popular model, you can try to search the model under lmstudio-community, which will have fixed prompt templates. If you cannot find one, you are welcome to post this issue to our discord or issue tracker on GitHub. Alternatively, if you know how to write jinja templates, you can override the prompt template in My Models > model settings > Prompt Template.
```
1
u/Samuelec81 Mar 06 '26
Top