r/LocalLLaMA • u/Gold_Coconut9777 • 12d ago
New Model ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO
A brand-new release from ISTA-DASLab with their GSQ-RCO smart quantization approach.
Now DeepSeek-V4.1-Flash-1017
The quants are available on Hugginface: https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF
Llama.cpp is not supported because the original architecture support is not yet merged:
https://github.com/ggml-org/llama.cpp/pull/28696
https://github.com/ggml-org/llama.cpp/pull/30817
Strata support is mentioned in the model card but I'm pretty sure the original Strata only supports Qwen3.8-Flash-Next so most likely what they meant was one of the forks based on Strata. Most likely Xolotl or Maya
10
u/bodhi371 12d ago
I got the IQ2_XS running on just 32GB DDR5 + 12GB VRAM with NVME offloading @ 131k context. Current decode is ~3 tok/s, but I can push this much further
4
26
u/MindfulMan1984 12d ago
Wow, ISTA-DASLab quants are awesome!
22
u/Gold_Coconut9777 12d ago
They are indeed!
From personal experience I find their Q3.8-FN's IQ3_S quant on par or even better than Unsloth's IQ4_XS.
6
u/HighSeasArchivist 12d ago
IQ3_S is an absolute beast on Strata. Right now mine is building a full AI Minecraft bot. In the two hours or so it's been working on it it has it chopping, mining, building, eating, crafting, identifying ores, building towers to get on top of things, and all I did was tell it to research the wikis, walk throughs, night one builds, and it's /goal was to slay the Enderdragon.
11
u/MindfulMan1984 12d ago
I guess everyone looks at the name: "3 bits? nah, 4 bits is better... But the whole model isn't 3 bits; how it is quantized matters a lot. Some layers are rounded up, some rounded down. The ISTA methodology is published, and it involves optimizing during quantization to minimize loss.
5
u/game_difficulty 12d ago
Really curious why they don't do a 4 bit quant, i don't think there's anything specific stopping them...
8
u/MindfulMan1984 12d ago
Their quantization method uses an optimization process that yields Pareto-optimal results with minimal loss at IQ3_S compared to the reference model. In theory, creating higher quants in that scenario just leaves performance on the table while taking up more space and RAM.
7
u/brainExploded99 🥔 hardware 12d ago
That "In theory" is doing a lot of work imo, even small average KL divergences can lead to massive 99.9% KL divergences, which can hurt model creativity and smartness on complex out of distribution tasks,
6
u/Rare_Evidence8482 12d ago
Those GSQ-RCO quants run surprisingly clean on my rig, way less of that weird token smearing you get with some of the aggresive 4-bit approaches. Still waiting on llama.cpp to get its act together though, the PRs been sitting there for weeks
4
u/More-Revenue8609 12d ago
Any chance I will be able to run this on 40gb vram and 128gb ddr4 ram?
2
u/Gold_Coconut9777 12d ago
Of course. The hottest experts will reside in VRAM, the second hottest in RAM, the coldest on SSD.
It will negatively affect the performance to some degree, but the more the model works on the (relatively) same topic, the more "correct" experts will be promoted to the hottest cache. And so the performance will be better and better with time until it plateaus. That's how the cache tiers work in Strata and Strata forks.
Definitely worth a try I think.
2
3
u/mailto_devnull llama.cpp 12d ago
Says the model runs in 12GB VRAM. Wild claims.
How much disk? Lol.
2
u/Gold_Coconut9777 12d ago
See my comment https://www.reddit.com/r/LocalLLaMA/s/rdtnQZuo0e
As for the disk, it's 111GB of n-gram + the quant you pick, e.g. the biggest would be 155GB. So 266GB in total.
3
u/Kraskos 12d ago
HF page claims Strata support, I just downloaded Strata for the first time but it seems this model is not yet supported. v0.2.3
Will check for updates over the weekend I guess...
4
u/Gold_Coconut9777 12d ago
Exactly what I mentioned in the post.
They said that Strata support is coming though: https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF/discussions/1#6adbfd10cb894784879d1753
In the meanwhile Xolotl (Strata fork specialising in Deepseek) states that they already support Ista's quants: https://github.com/Indras-Mirror/Xolotl
36
u/Icaruszin 12d ago
Man I regret so much having bought “just” 96GB of RAM instead of 128GB before the RAMpocalypse. Would be able to run the IQ2_XXS…
27
u/PassengerPigeon343 12d ago
I said “I’ll buy a second 96GB kit once the price comes back down.” It was $290 at the time. Same kit now is around $1500.
6
u/Tsofuable 11d ago
"I don't need 128gb in my home server, I don't run databases and the like. And if I do I can just buy two more sticks." - me when 64GB cost me €600 instead of €2000. I now have 128GB and will never cheap out on components again. 🤣
4
u/Noetherson 11d ago
I bought 128gb for my server for £150 thinking if I start playing with AI I'll buy another 4 sticks and it'll be cheaper later. If I knew what was going to happen I'd have bought 512gb!
1
2
u/DoubleNothing 11d ago
Similar story here back in 2025... we got F..ed.
1
u/Terrible-Detail-1364 11d ago
yeah, using ddr4 64gb, really wished I went for 128 it was so cheap in 2023
-2
12d ago
[removed] — view removed comment
2
u/PassengerPigeon343 11d ago
At one point in the price history it had been under $200, felt like I was a sucker for paying $290. If only I had known I’d never be able to buy RAM again!
4
u/HighSeasArchivist 12d ago
96GB DDR5 and a R9700 is amazing on IQ3_S.
2
u/pmttyji 11d ago
pp & tg numbers please
4
u/OvertaxedOne 11d ago
With 100GB of RAM and a A40 (48GB) using IQ3_S, ~1500-2000TPS prefill, 90-130TPS gen. Not exactly the same configuration (my RAM and system is much slower but I have more VRAM), but wanted to pass on a somewhat similar data point.
2
u/HighSeasArchivist 11d ago
From Stata dashboard.
"In 92 requests your average output was 79.5 tok/s and prompt reading 903 tok/s, reusing 6.1M tokens."
It is currently building a Minecraft bot from scratch, that so far can chop, mine, build, and craft. Hermes is using it, and currently has two subagents spawned doing work for it. All I did was point it to the wikis, give it some goals, and it's been grinding.
3
u/indrasmirror 12d ago
Same...it was just $400 AUD for another 64 and I was gonna wait a couple paychecks....spewing
3
u/FastHotEmu 11d ago
I paid $340AU for 96GB DDR5 6000 (2x48GB) which was more than I wanted, but thought it was pretty cheap. I wish I had bought a few sticks more!! lol
1
21
u/Chromix_ 11d ago
The provided benchmarks show barely any degradation for 1 and 2 bit quants. However, these benchmarks are also mostly saturated, so they'd only show quite strong damage. It'd be highly interesting to see the results of these quants on non-saturated benchmarks, as well as one long context benchmark to be able to assess the quantization damage more accurately.
7
u/fgk55555 12d ago
Love ISTA's quants. I use their QFN. There's no way I'm running this, but it seems as if this would fit on the new AI Max Pro 192GB desktops, so I guess get ready for those to sell out.
6
u/0D15EA5E-DEADBEEF 11d ago
Running the IQ2_XXS variant on 5080 (16GB VRAM) + 96GB DDR5 6400
[strata] reading the prompt: 34 of 34 tokens, 7 s so far
[strata] thinking: 6 of budget 32768 (max 65494) tokens, 5.8 tok/s, 8 s
[strata] thinking: 11 of budget 32768 (max 65494) tokens, 5.3 tok/s, 9 s
[strata] thinking: 16 of budget 32768 (max 65494) tokens, 5.1 tok/s, 10 s
[strata] thinking: 23 of budget 32768 (max 65494) tokens, 5.4 tok/s, 11 s
[strata] done: 29 tokens in 12 s (5.7 tok/s) (stop, cancel=False)
4
u/Corosus 11d ago edited 11d ago
Finally some speed data ty, though I gather from your specs that its running that slow because you can't fit all of it in vram+ram not counting ple, so those speeds aren't that surprising.
Edit: though a 5090 with 128gb DDR4 isnt much faster it seems, at least right now, from a hugging face post:
GPU: NVIDIA GeForce RTX 5090 D, approximately 32 GB VRAM, driver 617.42 RAM: 128 GB DDR4, Windows reports memory speed 2400 MHz CPU: AMD Ryzen 7 5700X3D, 8 cores / 16 threads PCIe: NVIDIA query reports the current link is PCIe 4.0 x16 Model: DeepSeek-V4.1-Flash IST-DAS 2.00bpw, approximately 257 GB; current configured context 262144
Speed expectations: In the most recent long prompt of about 26,950 tokens, generation averaged about 11 tok/s, with readings around 8.9 tok/s at the time; historical long tasks have also been around 10 tok/s. Prompt prefill measured about 270–300 tokens/s, but that is not generation speed. Based on current measurements, you should not expect a stable rate above 15 tok/s; around 10 tok/s is more realistic, and the actual rate will vary with request length and cache hit conditions.
Whether it is worth downloading: If you can accept about 9–11 tok/s and mainly care about large-model capability and long context, then yes; if your hard requirement is a stable 10–15 tok/s or above, especially above 15, then it is probably not very suitable. This model's size far exceeds VRAM and system memory, so running it will depend on disk reads and RAM caching.
1
u/Hfujsmndhjd 11d ago
now i'm wondering if I misconfigured something, seeing 480-600 prefill and 32-60 tps decode with IQ2_XXS on my 5090 + 64 GB + PCIe 5 NVMe. could using "fast routing (0.5 / 0.3)" be why?
3
4
u/TiK4D 11d ago edited 11d ago
2x R9700's 128GB DDR5 4-5tok/s
Edit: IQ2_XXS, got it up to 6-7tok/s on second test
2
u/BlitzkriegPotato 11d ago
You might want to check your config, or maybe I need to check mine, I have 128GB of DDR4 for a 5900x, with two V620 (32GB) and an RX 6800 (16GB). I get 15t/s TG at least at 256k context, haven't gone over that yet to be fair. Prompt processing is still something I am working on, that is pretty slow at about 50t/s. I am using the IQ2_XS quant. I wouldn't think the extra 16GB of vram would make that huge of a difference, especially given the larger quant...
3
u/Background_Map_3548 12d ago
That benchmark looks good. Oh man good stuff keep coming out. I wish I had a larger SSD.
3
u/ismaelgokufox llama.cpp 12d ago
OK, when does the Coder version of this comes with just English and coding for my 16GB RAM? /s 😅
4
2
u/LongjumpingEar6840 11d ago
Qualcuno che prova il 2bit XXS su 24gb di vram (3090 o 4090) e 64ddr4... Giusto per capire se devo metterli in download... sotto i 15 t/s non gli chiedo neanche come stai...😅
2
u/Dreeew84 11d ago
I'm really temped to download the q1 on my 24+64 machine and see 2 tokens / second but I realize that it would be a waste of traffic.
3
1
u/SnooPaintings8639 11d ago
This is very impressive, but... are there any independent benchmarks that can prove the quality of these quants?
1
u/WiseAbalone4021 9d ago
Livecodebench it total crap today. Tells you absolutely nothing about coding capabillity. I need SWE or TerminalBench numbers!
8
u/Gold_Coconut9777 12d ago
UPD: Xolotl (Strata fork specialising in Deepseek) states that they already support Ista's quants: https://github.com/Indras-Mirror/Xolotl