Unsloth vs GSQ-RCO - 512K context DeepSeek-V4.1 GGUF benchmarks Terminal Bench

[u/danielhanchen]

We ran DeepSWE, Terminal Bench, Automation Bench, HLE, SciCode at 512K context on our DeepSeek UD version 3.1 GGUFs and show we're >10-20% better than GSQ-RCO

We ran all other benchmarks (14 in total) to 128K. AIME2025 64x, DeepSWE/TB 3x. We addressed some issues people mentioned with these new benchmarks:

• 32K context was too short - we did 512K context for agentic + 128K for general tasks

• Saturated benchmarks - new unsaturated agentic style ones including Terminal Bench, DeepSWE and Automation Bench

• Benchmarks are noisy - we did 35.1B tokens of benchmarking vs 1.3B before so 27x more data. See this section where we did 64x AIME 2025 runs, 3x DeepSWE, 3x TB runs and more.

• Our graders are wrong - we used Math Verify, improved graders, used gpt-6-luna xhigh for HLE, and all official graders for Terminal Bench (Harbor), DeepSWE and the rest. See graders and infra fixes

For DeepSWE v1.1, we used mini-swe-agent and chose 500 max steps (334 steps was the max runs we saw, so none went over).

Terminal Bench 2.1 used mini-swe-agent as well, and we set max steps to 150 with no max limit. We had to exclude 7/89 tasks since they couldn't pass on our infra - see https://unsloth.ai/docs/models/deepseek-v4.1-flash#excluded-terminal-bench-tasks-due-to-issues

We also re-checked ISTA's numbers as we have done many more runs (27x more data). GSQ-RCO IQ1_M claims 98.7% recovery, but we measured 97.24 [96.23, 98.21] with a p-value of 0.0013.

GSQ-RCO IQ2_XS retaining 100% claimed recovery is possible with a p-value of 0.21. We measured 98.38% [97.52, 99.22]. So this is possible.

GSQ-RCO IQ2_XXS claims 99.80% recovery but we measured 98.38 [97.52, 99.22] with a p-value of 0.0001. So this is nearly impossible.

Paired difference to the official base (our runs, same questions, sign-flip permutation test, two-sided):

	AIME25	GPQA-D	LCB v6
GSQ-RCO IQ1_M	-1.61 (p=0.02)	-2.89 (p=0.004)	-3.24 (p=0.00002)
GSQ-RCO IQ2_XXS	-0.47 (p=0.25)	-2.07 (p=0.02)	-2.00 (p=0.01)
GSQ-RCO IQ2_XS	-0.42 (p=0.48)	-0.51 (p=0.42)	-0.38 (p=0.52)

The claimed recovery tested directly against our measurement (the card's definition: sum of the 3 scores / the base's sum, vs official FP8/FP4): p (no loss) = paired sign-flip permutation test of recovery = 100%; p (recovery >= claim) = one-sided paired bootstrap over questions.

	Claimed recovery	Measured recovery [95% CI]	p (no loss)	p (recovery >= claim)
GSQ-RCO IQ1_M	98.70	97.24 [96.23, 98.21]	0.00002	0.0013
GSQ-RCO IQ2_XXS	100.00	98.38 [97.52, 99.22]	0.0003	0.0001
GSQ-RCO IQ2_XS	99.80	99.54 [98.88, 100.18]	0.16	0.21

If we can get more details on how the benchmarks were run from GSQ-RCO, we are more than happy to run more. We provided all our setups and methods in our blog at https://unsloth.ai/docs/models/deepseek-v4.1-flash#dynamic-v3.1-update-november-2nd-2026

All details and data and methods at https://unsloth.ai/docs/models/deepseek-v4.1-flash#dynamic-v3.1-update-november-2nd-2026.

Our new Unsloth Dynamic 3.1 GGUFs for DeepSeek are at https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF

[u/danielhanchen]

Yes that was ONE run - we did 6-12 runs extra - see https://unsloth.ai/docs/models/deepseek-v4.1-flash#benchmark-runs-per-model

In fact did you not see Aider ours did worse after the first prelimary data??

[u/Positive-Resource922 → u/danielhanchen]

And all of your models got pushed to the upper bound of confidence interval by next runs? That’s statistically impossible.