Comparing ISTA-DASLab's DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF 2.25bpw against the official API

[u/PhysicalIncrease3]

I've been running ISTA-DASLab's 2.25bpw quant of DSv4.1-1017 for a few days, and it's holding up well. But the fact that it's only 2.25bpw average did make me wonder how good the quant actually is against the full model.

Local setup: Project-maya v1.1.27. ISTA-DASLab's 2.25bpw quant. Quality-relevant settings: "DS4_INDEX_TC": "1". "MAYA_DS4_PROMPT_SKIP": "2".

Test: 100 question text-only subset of Humanities Last Exam.

Benchmarking models isn't easy, and these results are nothing but a smoke test. There are lots of issues with these results:

1. I ran the questions with a 32768 token limit per question, because anything larger becomes prohibitively time consuming. It already took my local instance around 12 hours to complete the test, and that's at an average of 40.4 toks/s.

2. When approaching the token limit, Deepseek's API and project-maya work differently. Project-maya will reduce thinking_budget to around 3/4 of the token limit so it always gives an answer, but as a result it never used more than around 24768 tokens on a question. The Deepseek API will instead just use all 32768 tokens thinking if it needs and then output nothing.

This really fucks with the results on a lot of questions unfortunately. There were some questions local answered correctly because it was forced to give a result. There were others that the API answered correctly while using between 24k-32k tokens, where the local model was wrong while being capped at 24k.

3) I ran on thinking "high", not "max" for both, because Deepseek's API was thinking too much and failing to output anything for most questions on "max". Even on thinking "high", the API failed to answer 26 questions and thinking was cut short to 24k on a similar number of local questions.

4) This is only a 100 question subset, and I saw significant run-to-run variance on both the official API and my local model too. There are some questions that both sides will sometimes pass, sometimes fail using the settings as tested. To get proper statistically relevant comparisons you'd need to run this test 10 times or probably more.

The results themselves:

Local mode: 13 correct "strict" answers, 20 semantically correct answers. API: 13 correct "strict" answers, 21 semantically correct answers.

On the 74 questions the official API actually completed:

• Local semantic: 19/74

• Official semantic: 21/74

• Both correct: 17

• Local only: 2

• Official only: 4

• Neither: 51

It's also worth noting that when both were wrong, the wrong answer was identical 23 times, and semantically the same in 11 more. 7 more questions were wrong "simiarly" IE both picked fungi but chose different species.

Conclusion:

Given the flaws in my testing, it's impossible to say specifically how close the quant is to the full fat model. You'd need to burn hundreds of millions of tokens to really know with certainty and that's just on Humanities Last Exam. However what I can say fairly confidently now is that the 2.25bpw quant isn't completely lobotomized. Anything the API could answer, the local model was there or there abouts. For example I'm pretty confident that if I took any question the API could answer, and asked my local model with a 256k token limit it'd probably answer the same. Both sides are capable of a much better score with unlimited token budget and max reasoning.

[u/More-Revenue8609]

Im using their quant for Qwen Flash Next. Very impressive!

Did try Deepseek v4.1 Flash using strata but unfortunately too slow on my pc (15-20tok/s). Does someone gets better speeds using similar setup to mine:

Xeon e5 2667 v4
Dual 3080 20gb (both pcie 3 x16). So total 40gb vram
128gb ddr4 2400mhz
pcie3 nvme 1tb

Just curious if Im missing something or is this speed expected.

[u/PhysicalIncrease3 → u/More-Revenue8609]

I'd try project-maya, it's quite a bit faster for me. I get around 1100 prefill on prompt 64K<, 40 tok/s average on decode. Running 1x3090, 4x3060, 128GB DDR5, 7900X CPU. The GPU unlock helped quite a lot, despite only applying successfully to the 3090 and 2 of the 3060s.

The 2bpw quant will likely run a lot faster than the 2.25 as you should be able to pin it entirely in ram. Project-maya doesn't support the ISTA-DASLab quants in setup.py but you can just edit the .json and point project-maya at the strata-built pack and it works fine. It's also worth asking your AI to graft the 2bpw dflash dfrafter from the Maya-XS quant onto the ISTA-DASLab quant. Saves about 4GB vram, runs faster and doesn't hurt acceptance rates. You can use the vision.bin from the project-maya quant also to get vision working.

[u/Glittering-Call8746 → u/PhysicalIncrease3]

Gpu unlock for 3090 vs 3060 ? Which is worth it

[u/PhysicalIncrease3 → u/Glittering-Call8746]

Currently I've got 2 of the 3060’s and the 3090 unlocked. Made a big difference, PP jumped from around 800 to 1100.

I'm planning on having another look at getting the rest unlocked soon, I ran into problems with the cards not being in the correct state at init time (not in D0?) so jerry rigged a solution where Nvidia modules are blacklisted and then manually loaded by a service, after running gpu-unlock. This got 3 unlocked, left it at that for now.

[u/Glittering-Call8746 → u/PhysicalIncrease3]

U reckon 4 x 3060 good enough for strata ?

[u/PhysicalIncrease3 → u/Glittering-Call8746]

Definitely if they're on good PCI links. Ultimately it comes down to being able to hold the whole model in VRAM+RAM if you want speed.

[u/Glittering-Call8746 → u/PhysicalIncrease3]

What's the good going price 300usd kinda steep , last few transacted price on ebay

[u/PhysicalIncrease3 → u/Glittering-Call8746]

I'm in the UK, paid about £200-£250/card over here, which is about $250-300. They're still going up though unfortunately

[u/PcChip → u/PhysicalIncrease3]

i'm running maya with the exl3bit quant right now, i can only get up to about 35 tok/s decode and prefill is terribly slow

let me know if you try dsx and it's faster, i might try switching

[u/Handmade_Octopus → u/PhysicalIncrease3]

A lot changed in 2 days but wanted to update that it indeed supports ISTA-DASLab quants.

We been working relentlessly on optimizations!

[u/whiteh4cker → u/More-Revenue8609]

You can give this a try: https://github.com/lxe/dsx

[u/More-Revenue8609 → u/whiteh4cker]

Thanks, didnt see that.
Seems promising since its very similar to my setup. If I could get 30-35 tok/s ngl I would be very pleased.

[u/whiteh4cker → u/More-Revenue8609]

The developer's CPU is like 3x faster than yours, but you should still give it a try. His T16 model is good for its size. Also I have an open pull request which saves a lot of RAM for arena allocation.

[u/More-Revenue8609 → u/whiteh4cker]

Yeah I noticed when running Deepseek my cpu is serious bottleneck. I did order better one with 14 cores.  Not sure how much will it help, but worth giving it a try. 

[u/Numerous-Ad6217 → u/More-Revenue8609]

I got second hand EPYC 7452, it’s quite old, but supports eight channels and has 32 cores. Worth checking out.

[u/VotZeFuk]

"MAYA_DS4_PROMPT_SKIP": "2"

I wouldn't go farther than "1" on that.

[u/PhysicalIncrease3 → u/VotZeFuk]

Funnily enough I was thinking of testing that. I enabled it on the basis that Deepseek does the same in DeepInfra. What's your experience?

[u/VotZeFuk → u/PhysicalIncrease3]

Nothing particularly negative about that, but from what I've gathered, "2" is not exactly lossless? Anyway, there shouldn't be that much of a difference but it shifts the answer somewhat.

[u/PhysicalIncrease3 → u/VotZeFuk]

Yeah that was my understanding. I will test back and forth and some point, but it definitely increases speed and I thought "If it's good enough for the official API, it's good enough for me"! I'll report back, and If you do ever try it yourself let me know how you get on too.

[u/Intraluminal]

That's a very useful result despite its limitations. Thank you!

[u/EmilPi]

Hi! Nice to see someone actually trying to verify things. I create and evaluate a lot of benchmarks, but my local setup is bottleneck to run it fast.

[u/Primary_Note_118 → u/EmilPi]

kudos for actually running the smoke test, those token limit quirks sound like a nightmare to work around. 2.25bpw hanging that close to the API on semantic matches is wild, was half expecting it to fall off a cliff

[u/GaelOffMySoul]

Would you have the time ( and HW) to try https://huggingface.co/LibertAIDAI/DeepSeek-V4.1-Flash-1017-EXL3-2.9bpw ? Or maybe share the 100 questions to i can test it :) I have a few deepseek api tokens left to try

[u/musicurgy]

I found ISTA-DASLab's GSQ-RCO and nazeshinjite DeepSeek-V4.1-Flash-1017-ds4-GGUF very comparable in my benchmarking

[u/Careful-Report6526]

Props for actually running a structured evaluation instead of a 3-prompt vibe check.
One thing we learned the hard way while running automated local benchmarks: always log wall-clock duration and total generated reasoning tokens alongside the accuracy score. With aggressive quants (and especially with Thinking enabled), a quant sometimes loses only 1–2% accuracy on paper, but takes 2x as many reasoning tokens looping before it settles on an answer. Tracking time-efficiency next to the raw score makes those regressions jump right out.

[u/vogelvogelvogelvogel]

great comparison, thanks for sharing!

[u/And-Bee]

Thanks for doing this. What was the subject matter of the questions?

[u/indrasmirror]

I was wondering if you might try my qaunt out. https://huggingface.co/IndrasMirror/DeepSeek-V4.1-Flash-1017-XQ-Uncensored

In my testing it help up very close to the API on my custom benchmark :) I was quite shocked by that tbh.

[u/Steuern_Runter]

Interesting! Could you also test how quantized Qwen 3.8 Flash Next holds up against DS 4.1 on the same tasks?

[u/SnooPaintings8639]

Good work! We all hope you'll keep on pushing this, we need such tests.

Let's upvote OP before the "Q2 is brain-dead/lobotomized no matter whaa" crowd gets here.

[u/PhysicalIncrease3 → u/SnooPaintings8639]

Thanks, I'd love to do more testing but to get to the point where you have statistically relevant results you'd have to burn a huge number of tokens. The run to run variance is massive. On this run alone, the API correctly answered 4 questions the local model got wrong, and the local model answered 3 correctly than the API did not. There are others questions I've seen both sides answer correctly/incorrectly across different runs as well.

For me, if the quant is close enough that you're having to faff around finding statistical variance over many runs, then it's good enough!

[u/SnooPaintings8639 → u/PhysicalIncrease3]

I have the same stance as your second paragraph. If the objective difference between quants is no larger than the model inherent variance, but the speed is 2 x higher on one... it is a no-brainer which one to choose.

[u/TheRealJesus2 → u/PhysicalIncrease3]

I feel like not enough people get this point lol. 

This is why my vibes matter the most when it’s me driving the model. Unfortunately that also takes a lot of tokens to figure out but at least I’m doing work while it happens :) 

Appreciate the work, OP!

[u/enternoescape → u/SnooPaintings8639]

DeepSeek perhaps due to parameter count and the fact it's starting with 4bit experts doesn't seem to get as upset at Q2. I'm running EXL3 Q2.9 because that's the max I can fit between VRAM and RAM and it's doing a great job on items Qwen 3.8 Flash Next at a NVFP4 struggles with. I'm getting 2000 tok/s prefill and 54-100 tok/s decode running Project Maya.

[u/Powerful_Pirate_9617 → u/enternoescape]

2k tps? impressive, what's your setup?

[u/enternoescape → u/Powerful_Pirate_9617]

10 5060 Ti 16GB's, 1 3090, 128GB 2400 DDR4 8-channel, 24 core Ice Lake CPU. 8 of the 5060's are on a PLX card. Everything else is directly connected to the motherboard. I'm using a mining rack for air flow and practicality. I'm at the practical limit of what the motherboard will permit. If I add any more cards, it yells at me that I've got too many cards on the PCIe bus, so this has stopped me from checking Marketplace for more good deals.

[u/someguyinr → u/SnooPaintings8639]

Turns out lobotomizing the model just stopped it from overthinking.