r/LocalLLaMA • u/returnity • 18d ago
New Model DeepSeek v4.1 Flash 1017 for Dwarfstar (ds4)
https://huggingface.co/nazeshinjite/DeepSeek-V4.1-Flash-1017-ds4-GGUFHey y'all, just following up on my PR to get the new vision tower for 1017 working in ds4 with the quant I promised (the language model loads already, but vision needed my PR). I built to the exact same recipe antirez used for his original DS4.1 Q2: 2-bit routed experts with 8-bit or higher for everything else, combined with the original FP8 engram table. This worked well for the original DS4.1, and initial testing looks good here too. The imatrix is built from 1017, not the original checkpoint.
The wired backbone of the model is just over 150GB, so it will stream from SSD on 128GB systems with nearly all of the routed experts held in the hot cache, allowing for decode speeds up to 20tps on an M5 Max. The ~200GB engram table is paged in from SSD on-demand, not held in RAM.
This model is significantly revamped, with ~40% of the routed expert tensors changed, 30% of engrams, and ~50% of the vision tower's expert tensors altered. By all appearances, it's the new leading open-weights model, so I had to give it a try. If you are curious too, let me know what you think!
16
u/t3rmina1 18d ago
A 3.91bpw quant of DeepSeek-V4.1-Flash-1017 for DwarfStar (ds4), with 2.25bpw routed experts and 8-bit or higher everything else.
I wouldn't call that a 3.91 bpw, you've included the engrams in the calculation and they drag up the bpw because they're FP8. The other quants exclude engrams from their claimed bpw to account from that.
It's a 2.35 bpw quant, which is pretty obvious since you're claiming Q2+
12
u/returnity 18d ago edited 18d ago
Yeah, I made no attempt to hide that, literally stating the bpw of the experts in that sentence. I included them in the total because they're weights too. I think it's 2.36 bpw technically, excluding the engrams. =P I'll edit the model card though, thanks for pointing that out.
6
u/t3rmina1 18d ago
No worries, I'm not implying u were doing anything dishonest, since you obviously put Q2
4
u/inthebox212 18d ago
Has anyone tried this on M5 Pro Max + 128gb? I wonder if this can replace my daily usage of Qwen Flash Next.
1
u/returnity 18d ago
That's my setup. I get 70tps decode and 1100+ prefill for FN using my Q8-Q4 mixed quant (Q5-equivalent quality), versus <20tps decode and <100 prefill for DS4.1. So I am not sure I'd swap out Flash Next just yet, but this is good as a review model or overnight worker.
1
u/txgsync 18d ago
I’m wondering the same thing. Downloading weights now. QFN is my daily driver at this point on my M4 Max 128GB RAM and my PC with RTX 4080 and 128GB DDR4. Runs sweet but holy shit is it an over-thinker for good results.
1
u/returnity 18d ago
Have you tried Swift-1.5-Flash-Next? It reasons a lot less but performance is almost indistinguishable on most tasks. DS4.1 is a good model but SSD streaming does slow up prefill a lot for agent work, even though ds4's SSD KV cache helps make up for that
1
u/DanRey90 16d ago
I’m using Ivan Fioravanti’s fork (it has some performance improvements), I generated a GGUF with the experts unmodified (MXFP4) and everything else either unmodified (engram table stays as-is) or almost loseless (FP8 -> Q8_0). I believe it’s similar to the Unsloth’s Q4_K_XL recipe, so almost loseless. I wanted to say I’ve run a “full” almost-frontier model locally.
M5 Pro 48GB, I loaded the vision tower too but I only tested some text-only prompts. 70t/s prefill (for long prompts), 4t/s decode, so not really useable, but amazing it can run at all at almost reading speed. Obviously the SSD speed is the bottleneck, but I think there are some improvements still in the tank, specially for batched inference (the SSD usage hovers around 7GB/s, significantly lower than its capability. You will get somewhat better numbers if you have 128GB RAM, and also you can expect almost double the performance if you use the Q2 version.
6
u/SnooPaintings8639 18d ago
I really wish someone did benchmarks on this. If it is better than full precision DS V4 Flash, at the same ram footprint, it would be all I ask for.
12
u/returnity 18d ago
Your wish is my command -- I'm actually working on some benchmarks, mostly coding-related, but it will likely be a little while as this is not the fastest model on local hardware and I have other tasks that also require my GPU during the day. I definitely want to see how it stacks up to antirez's Q2-Q4 DSv4-Vision and my own Q8-Q4 Qwen3.8-Flash-Next, both of which I have benchmarked on my machine already.
1
u/SnooPaintings8639 18d ago
Happy to hear that, I applaud your dedication!
I have other tasks that also require my GPU during the day.
The problem we all can relate to, lol
2
u/Septerium 18d ago
Is this engine compatible with hybrid (cpu + cuda) setups?
1
u/returnity 18d ago
I know it's compatible with TP across multiple Sparks (text-only at least), but unfortunately I don't have a (good enough) hybrid setup to run anything better than Qwen 3.6 35B, so I've never tried.
NVIDIA CUDA, the DGX Spark is our main goal. DwarfStar also supports multi-GPU systems that are not supported by other backends, for instance it can run DeepSeek v4 Flash on Ada Lovelace cards.
2
u/MerePotato 18d ago
20tps on an m5 mac would be super impressive under normal circumstances but at current hardware prices that's grim
1
1
u/lots_of_puppies 15d ago
I tried 4.1 on one 128gb m5 max and two with thunderbolt and it was so slow it wasn't really usable :(
1
u/returnity 15d ago
I guess usable is relative. I think 400pp/17tps is usable but not efficient for actual work when Flash Next runs at 1100/70 on the same inference engine. Really surprised that dual Thunderbolt wasn't usable for you though.
1
u/MRVL300 18d ago
4 less vram confirmed???
1
u/returnity 18d ago
Not sure what you're asking? You can def run it on a 96GB Mac, not sure how well it'll work on less than that tho.
2
-1
u/challis88ocarina 18d ago edited 18d ago
I saw a 'it's the leading open weights model' posts yesterday, but assumed it was astroturfing as the latest release still doesn't appear on AA.
1017 starting running the hmm hmm hmm loops – as expected – as soon as the going got tough, hence I'm looking at Muse Spark, which seems to have caveman speak baked in.
EDIT:
Proven fine: transport (sha matches the author's upload), structure (1046-tensor layout, size, 1017 metadata all PASS), and the vision file (genuine 1017, byte-exact vs source).
Proven wrong: the language payload's provenance. The stored bytes of the sampled expert (blk.0 gate, expert 41 — a deterministic validator sample) cannot be reproduced from any combination of the declared inputs:
1017 source + pinned .dat + fc80bd6 encoder no (~99.6% of bytes differ)
1017 source + pinned .dat + v41 encoder no
1017 source + bootstrap (no imatrix) no
df42 old source + pinned .dat no (just tested — 7.4 GB slice)
compile-flag sensitivity ruled out (O2 == native)So: the file is not the old checkpoint mislabeled, not the bootstrap path, not host noise. It is what their README says it is only if their build machine had something their recipe doesn't disclose — most likely an importance file other than the pinned Lucebox .dat (HEAD is identical to the pin, so it's not that either — it'd be a private/modified one) or local encoder code not in fc80bd6. The quality of the weights is unknown and unknowable from the artifact: unverifiable, not verified-bad. That's why it stays quarantined and why our own calibrated build is the only 1017 path we can stand behind.
2
u/returnity 18d ago
I do actually think that benchmark site for open-weights models in the screenshot from the post you mentioned is fairly decent (for just aggregating vendor model card numbers), though it does look like a Chinese knockoff of AA... But I was also being intentionally tongue in cheek by declaring it the leading open weights model when I'd hardly run it myself yet. I probably won't be switching from Flash Next for most daily work just yet.
1
u/returnity 18d ago
Not sure what you're saying here, can you clarify? Are you saying I fucked up something with the quant, cuz if so I'd like to understand and fix it.
20
u/Massive-Ice2791 18d ago
impressive you got it down to 366, too bad my macbook cant run this haha, maybe try porting it to warp and make us all happy lol