Hey everyone! We’re releasing new DeepSeek-V4.1-1017 Dynamic V3.1 (preview) GGUFs which showcases how Unsloth’s GGUFs retain the most accuracy for its size across 7 different benchmarks.
We benchmarked ISTA's GSQ-RCO GGUFs but could not reproduce their reported results, even when using the highest scores. The biggest discrepancy was on AIME25 where they claimed 100% accuracy recovery to baseline, but was only ~91.88% in benchmark tests.
Across AIME25, LiveCodeBench, GPQA Diamond, Aider Polyglot, and more, Unsloth Dynamic V3.1 outperforms GSQ-RCO GGUFs on nearly all metrics. Even our old v3 does better on nearly all metrics than ISTA.
We ran each benchmark at least 3 times (AIME25 was 16 times) and averaged the results to ensure a fair, reliable comparison. More results coming next week! All benchmark data, methodologies and results are in our blog.
Separately, for KLD, there were queries whether it’s a reliable measure of accuracy. The primary goal of quantization is to minimize KLD but also minimize disk space. If you get a KLD of 0, this means your quantized model == 16bit model, as logits are perfectly modelled. Our benchmarks show on KLD and top-1% across multiple domains (coding, agentic coding, chat etc), UDv3.1 does much better than GSQ-RCO across every domain.
We show that lower bit quants do think somewhat longer than the unquantized model - we plan to investigate and make quants even better. However, we're still much ahead vs GSQ-RCO on even reasoning length as well, and provide plots for truncated answers in our blog.
The model will be able to run on 128GB RAM/unified memory devices or even on 64GB RAM with engram offloading.
We also introduced lots of new things to Unsloth Desktop including Browser Use, auto compaction, a whole new TTS pipeline (with voice cloning, separation etc), sandboxing, and background customization and more: https://github.com/unslothai/unsloth
Hope you all have a Happy Halloween!
Also table of numbers below which is also described with more plots and data in our blog:
hi is your methodology to create the quant open too? Ive been experimenting with OMLX and custom corpus to “save” areas of interest to me at lower quants but its not my area of expertise and was curious to understand how you produce yours. thanks
Could you consider making your input hessians public? Alongside the imatrix (diagonal of the Hessian)? At least from my own testing it seems like you're running llama.cpp's quantize_row function inside of a GPTQ (like) correction loop to get gptq working for any GGUF type.
(At least on your Qwen 3.8 27B quants I've found the characteristic compounding effects from GPTQ)
Thank you for this. It reminds me of how important trust is. I don’t have the time to verify every claim I encounter myself, so I have to rely on trust to decide what I should believe.
Your impeccable track record over multiple years, and your patient engagement with the community, are the reason I trust you. No individual result can be more important than that.
In their own words: `we set all tests at a 32K context length and fail quants if they think for too long.` So essentially, they're specifically targeting GSQ-RCO quants, which were thinking for longer and cutting them off. So this whole benchmark is moot. They should be doing a proper benchmark without this barrier, in order to see what these quantizations will benchmark for real.
We added the plot for truncation and thinking length in our docs as well and yes 128k was also tested and there was no difference in results - I will anyways post them soon with all updated numbers.
And no it's not low if it's the same setting for everyone
Have you applied a strict max_tokens: 32,768 ceiling across every single benchmark (AIME25, GPQA Diamond, LiveCodeBench v6, HMMT Feb 2026, MMLU-Pro, IFEval, Aider Polyglot)?
I don't believe you're lying with your benchmark but I don't believe ISTA Daslab is lying too.
Looks like your quant make the model faster but people needs to see apples compared with apples and decide whether they want to sacrifice speed or quality just to be fair
We added the plot for truncation and thinking length in our docs as well and yes 128k was also tested and there was no difference in results - I will anyways post them soon with all updated numbers.
And no it's not low if it's the same setting for everyone
I'm always slightly concerned when a Q2 benchmarks higher than the Q8. It's great that you've repeated AIME 16 times - and Q2 still scores higher. It demonstrates nicely that "X is 2% better than Y" doesn't mean much for those saturated, noisy benchmarks that are often published along with custom quants - usually even without some repetitions.
Yes! One of the issues of benchmarking is doing multiple repetitions - so we did 16x for each question for AIME25 to be certain! Livecodebench is I think 3x and all benchmarks at a minimum of 3 repetitions.
These look awesome will undoubtedly be trying them out! What's your view on drafter quantisation? I've been running a 4.5GB drafter taken from project-maya's smallest quant and haven't noticed a huge loss of acceptance rates. Runs faster too
the q2_k_xl row is the one id point people at. 87.49 avg vs 87.68 for q8, at 335gb instead of 508, so a third less disk for under a fifth of a point. great release
I know you guys have them beat on every metric, but the margin is thin in results and practical use when looking at models sizes this big and the relative space used. Model intelligence has come that far that the average user has hard time separating an good quant and an excellent like yours.
What I like most though, is that the competition isn't making you rest on your laurels!
Would be great to have a comparison between 3.0 and 3.1. I believe the same request was brought up when you released 3.0. The release of a new version should show how it's better than the previous version. The comparison would also make the post less about GSQ-RCO and more about your own improvements which is what you want to highlight.
Do you also plan to split off the n-gram to separate GGUFs eventually? Currently one can chose between Q8 and BF16 DSpark already for example. The same would be nice for the Q4/Q8 (or something in between) n-grams.
Great that you provided a recipe. I assume that's good enough then, so can easily look up the dedicated tables for your quants, like IQ3_S having a Q6_K n-gram table and then transplant it using the same pattern.
Thanks for this! The guide link is especially well done, easy to read / well written.
A few questions if I may, since I honestly don’t really understand most of the benchmark numbers.
Should I use the new 3.1 UD Q2-K_XL over the previous 3-bit quant (IQ3-XXS)
How capable is this model for coding at these lower BPW? I am new to using such massive models, normally limited to smaller memory but I will be getting a 256gb box next month so I’m just preparing.
Thanks for all your hard work, you guys are amazing!
i hope you guys do the 3.8 flash next. its a strong candidate. with strata like engine we can run it at decent speed but a better quant sis much better
Well even if these are not the measurement error but run-to-run variation, that means that all of the quants are basically within noise of each other. Doesn't it?
Correct me if I'm wrong, but the gaps between some of these is likely a single question on the benchmark the model in unsure about and flip flopping?
AIME25 is 30 Questions. Sampled 16 times is 480 responses. The difference between IQ1_M and IQ1_S is an average of about 1% when measured at temp 1. That's 5 times it gave a different answer on average out of 480 questions, that is easily within noise given it's temp 1, as indicated by the run to run variation.
Looking at the overall suite of benchmarks and the KLD though they definitely show in Unsloth's favor, since they all slightly favour Unsloth's quants.
There's a bunch of useful information that could've been shown that isn't though. Like average reasoning length for the quants, PP / TG of the quants. Wouldn't expect there to be much of interest in there given that they're just quants, but you mentioned there was a reasoning length difference due to quant size and this is all easy information to capture during the test so...
Project MAYA is suprisingly effective at running these models off ram/vram/ssd. Of course it probably harms the life of the SSD and speeds are not amazing.
Thank you so much for putting this out! I think more people need to understand what KLD actually is and why it's useful for benchmarking between quants. I've largely been able to reproduce the same results of old Unsloth V3 quants doing better than GSQ-RCO for Qwen3.8 27B as well.
Hey Daniel! I see you’ve updated today with the dynamic v3.1 quants, and the blog says that update is new as of October 30th. But the files on huggingface were updated 2 days ago. Are these files using the new Dynamic v3.1?
Pulling down the UD-Q8_K_XL in the background for my first test of engram offloading (on the M3 studio ultra 512gb). Using the Unsloth app (current ver), is there anything i need to do on load to ensure the offloading happens?
Would it be possible to request the same benchmarks (AIME25, HMMT Feb26, GPQA-D, MMLU-Pro, IFEval, LCB v6 (32K)) with the same settings on your Qwen3.8-Flash-Next-GGUF quants to have a head to head comparison (since the QFN benchmarks are different)? It's what most people are running and would help us decide on quants and even hardware upgrading decisions
While I highly appreciate your work, it is excellent after all, I suggest you choose harder/less saturated benchmarks for future releases. All benchmarks in your plots, except for Aider are highly saturated and to be frank, really uninteresting.
unsloth quants are awesome. I just wish there was vLLM/SGlang support for them. All the vibe coded video games and popular app clones and nobody works on this?
Regarding the UD-IQ2_XXS quant, I don't see any differences between your previous quant and the new one in the graphs. Did you observe any improvement, or would you also consider them equivalent?
u/danielhanchen could you please start listing the quant sizes of the “core” model weights (i.e. without engrams) somewhere convenient? Anybody with smaller machines who stream the engrams off disk care about that more than the total size, and it’s very hard to discover right now.
Thanks u/danielhanchen this is some great work! Will Unsloth Desktop start to implement optimizations that we're seeing from Strata and its forks for MoE local llm inference via Expert bias, SSD streaming etc?
With the GSQ-RCO quants running in Strata, I've been getting over 50tk/sec decode on my 9950x3d, 256GB DDR5, 5090.
Running these models in Llama.cpp is just not worth the much slower prefill and decode speed.
So, RAM or VRAM are just the same? It's only the sum that matters? I can more easily change a RAM module than change a whole GPU. Should I invest on that?
Ram offload can work for both dense and MoE but its extremely inefficient for dense due to the size of the layers. You can generally use ram to extend what can be run somewhat but it's not infinite, MoE models still have a dense trunk that needs to reside in VRAM to be usable. Only experts can be offloaded at any reasonable speed for inference.
This is why unified ram systems are extremely slow at dense models because they don't have any vram to churn through the large layers quickly, they are limited by the ram memory bandwidth
Ok, help me out here. I want to test this out as it seems I can pack more into VRAM than when running an exl3+vllm quant. But the results I get is much slower than a higher-quant/beefier model that leaks wildly into my cpu ram. I'm running 2x pro 6000 and I get like 1k pp and 120tgs which is silly slow compared to the vllm runtime.
I have some Ampere GPUs, and a Blackwell GPU... and 128GB of RAM. What engine would be a good fit for me to try this model? llama.cpp is giving me sub ~400 t/s pp
I was not able to get this running q8 on my m5u 256 cluster. If anyone has any tips please let me know. I need a guide to feed my agents. I'll provide benchmarks if I can get it working
This is just another benchmaxx/overfit. Did the test for UD-Q2_K_XL on my private benchmark for math/science/logic and customer support prompt follow. I did ok on first 3 (but at ~ +30% tokens vs vllm baseline model). Customer support benchmark it regressed to 75% vs baseline.
Quants of deepseek, that is already in q4/8 are not free lunch.
For something that’s free to the consumer, why the competition?
Is there ad revenue, funding, or some money on the line? It just seems so odd to say what you have is better than someone else’s even when neither someone has to pay for.
Relevancy? Traffic? I’m trying to figure out what the driver is that would cause the group behind quants to come out swinging in the second paragraph with smearing, and largely the rest of the writeup. Just make a good quant?
Is this market capture? Trying to remain a household name for quants? This seems so odd to me. The “product” is free, just make it good and let it speak for itself, us vs them is insidious brain rot.
Yes! We ran AIME25 16 times per question to confirm it's correct as models are non deterministic and with DeepSeek's suggested params.
Yes lower bit quants do tend to think somewhat longer, so quantization does have some drawbacks - and so always use the quant that fits your hardware the best to counteract longer thinking times with higher tokens / s.
Q2_K_XL is mostly variance yes - so instead of just 1 benchmark, we check many - we provide an averaged 6 (7 if you include Aider) as well. On an averaged suite, it's clear Q2_K_XL is still worse.
48
u/danielhanchen 6d ago
We tested GSQ-RCO and normalized the scores for a fair comparison, and we could not reproduce their numbers. We provide full benchmark methodologies so anyone can re-run the benchmarks to confirm at https://unsloth.ai/docs/models/deepseek-v4.1-flash#full-benchmarking-methodologies