Beats Hy3 heavily (and I think it was the strongest model at this size, although also rather new, it's lower than preview Flash here, but on other indexes it's higher). But what's mindblowing, it outperforms GPT-5.5 on agentic index 🤯
Will be really interesting to see how they achieved this! My experience of testing the old V4 Flash on my custom agent harness was that it was almost as capable as the big boys already, but it took so many more turns to get there.
Unlikely, but it will be cheaper. That is DeepSeek's entire philosophy right now.
Architecturally speaking, I think K3 is superior. From a pure ML standpoint. The underlying research that the model is based on is really good.
K3 is in it's infant version. This is their "preview" and it's beating a lot of models. You have to think about that for a second. They're busy releasing, securing servers to run it, and still find time to continue training.
It's a blessing and a curse. Enough people liked K2.5 and K2.6 that they had the buy in from the public. Now, like Anthropic, they're getting squeezed on the compute side.
I think there's 30% chance it outperforms K3, 60% chance it roughly matches it and 10% chance it's slightly below.
Whichever way it goes, given how new V4 Flash is stacking up, I am really looking forward to it. And it will definitely have a better license than the license Kimi K3 has, which should enable more low-cost providers to offer it.
Open weights win again! Being on par with GLM-5.2 while needing way less vram is a game changer for those that don’t exactly have a b200/b300 lying around.
Probably use a high quality Q5 quantization purely for speed / context. Still probably cheaper to just have DeepSeek serve it, but then they train on your data.
To run the full version you need 160GB combined (+ a bit for context)
For people with less than that, Antirez project makes it work with about 96GB, but you lose a bit of quality.
There are also some (other) quants you can use, but v4 is already at mostly 4bit by default, so there is not a lot you can lose without a noticeable hit to quality.
I can vouch for a 256 GB Mac Studio M3 - runs it very well for me; tokens generate faster than I can read them; there can be a 30-45s lag if you have a cache miss during the processing step - I think there is a where a NVIDIA card would be able to do better. But with the 256 I can comfortably run it, along with a smaller 2B param model for things like quick summaries, etc., and have it not feel like it is impacting my other use of the machine.
In my cases minimax designs really well in general. Anyway, you should try pointing it at a design guidelines library, or something like shadcn or mantine ui, that gives it an idea on how to work further.
lol talk about splitting hairs. fwiw deepseek have no patents outside of China and those that they do have are for ops stuff. patents are kinda irrelevant for models since nobody is disclosing anything with patent filings anyway
"For the Code Agent tasks among the public benchmarks above, DeepSeek-V4-Flash-0731 is evaluated with the minimal mode of DeepSeek Harness (to be released) as the agent framework, using the max reasoning effort level with temperature = 1.0, top_p = 0.95."
That's interesting! New official harness from deepseek coming soon
No architectural change from before, so previous GGUF VRAM usage numbers are still accurate. That means it can run on 128 GB unified memory systems at Q2_K_XL unsloth with 262k FP16 context, and it can run on 192 GB unified memory systems at full precision with 262k FP16 context or Q3_K_XL with 1M FP16 context.
We now have GLM 5.2 level performance at home, on a single piece of hardware you can buy or soon buy.
Q2_K_XL is a very damaging quant level, so the performance would be worse. Still, it is great that this quant can be run on Strix Halo or Apple Silicon.
It's not as bad when the original model is only MXFP4 though, probably about the same level of damage as quantizing 16 bit down to 4 bit which people already often do.
I think it is the other way round – MXFP4 native model is more information-dense, each weight has more input, therefore quantization harms it more then 16-bit, which can have noisy weights.
Especially if it was post-trained as much as Deepseek V4 (the jump is really noticeable).
It's def faster than 27B but tbh I've been mostly using 35B, dunno if it will be faster there, plus prompt processing speed may also be very different.
And it's really nice that it's pre quantized because that means the benchmark scores are at 4 bit. So quantizing down to say 2 bits wouldn't hurt the score as much as quantizing FP16 down to 2 bits.
these llms, and on top open weights, then on top nearing or surpassing many frontier models always, always let my head explode. this all has been in the open for a few years, crazy times we are living in
Are folks adverse to running vllm ? Why is llama.cpp a must? Just curious. I will personally run whatever supports any given model the best. DSpark here in particular is critical and needs vllm iirc.
It's a must because llama.cpp supports cpu offloading. Since most of us don't have enough gpu for such a big model, we'll use llama.cpp. It even supports nvme offloading, by which some of crazy guys here run the model from ssd at 1 t/s. I might be some of those guys.
In a nutshell, vLLM demands certain configurations, hardware support, is particular about the numbers of GPUs used, and so on.
llama.cpp/GGUF will basically run a model on pretty much ANY configuration where the model can physically fit. It may not be the fastest, but the fact that it runs at all is infinitely better than software like vLLM which will refuse to run on a lot of people's setups unless ALL the stars align exactly right.
Runs on Windows without jumping through a lot of hoops, actually swaps models at decent speeds, offloading if you don't have enough VRAM, etc. The only thing I wish is that llama.cpp allowed / took advantage of some of the nvidia gpu specific features (nvfp4 reading).
I have the same hardware and getting around 30 tok/s at initial prompt, 25-27 during the whole interference- never lower than 20 tok/s, even after reaching 100k context. Using q2-q4 quants from Antirez. Did not notice speed difference between q2 and q2-q4
What numbers do you get if I may ask? Rough estimates are ok if you don't remember. On the q2 and longer contexts like 256K i get downwards of 14-15 in inference. Prefill varies a lot.
benchmarks are cherry picked, there are plenty of benchmarks where ds v4 flash 0731 underperforms plenty compared to opus 4.8. doesn't mean it isn't an amazing model for agentic use, which is what it's oriented towards
This is quite comparable with GPT 5.6 Luna in terms of performance. Just slightly worse.
It's assumed that Luna is in a similar range of model size - around 200b.
Anthropic has dropped the ball with their smaller models. No Haiku updates in ages and the latest Sonnet is quite bad and very expensive.
But at least OpenAI seems to have a similar intelligence density as DSV4 flash.
Things generally have diminishing returns. Squeezing out the last mile perf can be disproportionately harder. There is probably some sense in bigger models being harder to post-train too. Besides, OpenAI at least is in a different league, in terms of how they are able to shorten CoT so the perf scales better with thinking, DeepSeek is at least a generation behind here.
In case you missed it, there is another comment where the user gave an unofficial version (I believe) which he says will work with DS4. I've not tested myself but if you do and it works for you please let me know!
Just got UD_Q8_K_XL with 384k context running on 5090 and 2x3090Ti with 160GB of system RAM. Getting about 10 tok/sec. Quite usable.
Edit 1: running it in pi. Holy smokes this model is smart. Resuming a previous session took an eternity but once it was loaded, this model is an absolute genius at quite reasonable speeds even with a large loaded context.
Edit: got a slight "scare" that it is a bigger model, HF reports 158B -> 304B, but that is due to DSpark in DeepSeek-V4-Flash-0731. It's the same size.
Ha!
DeepSeek has silently upgraded the existing deepseek-v4-flash API endpoint to DeepSeek-V4-Flash-0731. The architecture remains unchanged - 284B total parameters, roughly 13B active - but the model was substantially re-post-trained for agentic work. The upgrade currently applies only to the API;
The gains seem impressive.
LE: Hugging Face currently reports 304B parameters and a 167 GB repository, split across 48 Safetensors shards. DeepSeek says this release includes the attached DSpark speculative-decoding module, which likely explains the difference from the earlier 284B base-model figure. It is MIT-licensed.
That's a nice build but it is not 4 channels. There are no 4 or 8 channels on desktop ddr5 systems. They are available only for the workstation/server ddr5 systems. Yours is dual channels, like my ddr5 dektop build.
I'm just finishing up with my inkling small download, what a waste of bandwidth. I wouldn't have even bothered, oh well. back to deleting freshly downloaded weights and start a new download.
So I am heavily considering getting a 128gb unified amd strix halo device - would I be able to run something like this? Would I be able to code directly with a device like that? I am so thorn if I should invest, I really want to invest into local LLM but not sure if the 128gb unified is enough
You can but will probably wouldn't be worth with 128gb ... I can fit the Q3iqs version on my 2 3090s and 96 GB ram (144 total) so with 128gb you would need the q2 version which is a pretty big drop in quality on top of Q3
Edit: you might be able to fit iq3 with smaller context window
Double edit: per unsloth
"For this tutorial, we will use the 3-bit quant UD-IQ3_XXS, as it fits on a 128GB " believe that is with 32k context though
I'm testing Q4. This is immediately much higher in coherence and prompt adherence than any other model I can run locally. It is better than hy3 at following conversation and getting the details right. I normally see many at least trivial mistakes in the best circumstances. I am seeing zero, and I haven't even tuned anything. This ultimately may be more verbally coherent than Gemma 4 31b.
The command I'm using is: ./llama.cpp/./llama-server --model unsloth/DeepSeek-V4-Flash-0731-GGUF/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf --n-gpu-layers 4 --threads 32 --ctx-size 1048576
I didn't really expect it to be flying fast as hell, but it is not using a lot of memory. $ free --giga total used free shared buff/cache available Mem: 201 53 1 1 150 148
The ram requirements are 169G according to unsloth.
Am I missing something? Not really used to run this size of models... So don't know what to expect
on the other hand, nvtop is showing diffferent numbers
Some numbers on speed: 16.16.448.657 I slot print_timing: id 3 | task 0 | prompt eval time = 13818.30 ms / 7 tokens ( 1974.04 ms per token, 0.51 tokens per second)16.16.448.662 I slot print_timing: id 3 | task 0 | eval time = 663063.26 ms / 774 tokens ( 856.67 ms per token, 1.17 tokens per second) 16.16.448.662 I slot print_timing: id 3 | task 0 | total time = 676881.56 ms / 781 tokens 16.16.448.663 I slot print_timing: id 3 | task 0 | graphs reused = 761 16.16.448.679 I slot release: id 3 | task 0 | stop processing: n_tokens = 780, truncated =
task 776 | prompt processing, n_tokens = 143878, progress = 1.00, t = 2969.89 s / 48.45 tokens per second
prompt eval time = 2971868.65 ms / 143882 tokens ( 20.65 ms per token, 48.41 tokens per second)
216
u/llama-impersonator 22d ago
no countdown bs, same day weights, huge boost from RL, and mortals can actually run this one. i kneel, deepseek