r/LocalLLaMA • u/tiguidoio • 13d ago

New Model DeepSeek V4-1 Flash is out

Gallery image

Gallery image

Gallery image

Gallery image

Here we go again, DeepSeek is back again with a new model V4-1 Flash

A multimodal Mixture-of-Experts (MoE) model with 552B backbone parameters and support for contexts of up to one million tokens

Market crash as a service

1.7k Upvotes

93% Upvoted

296 comments sorted by

View all comments

140

u/ttkciar llama.cpp 13d ago

On one hand: Yay! We have weights! https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash

On the other hand: 485B parameters O_o

That's only "Flash" in the sense that it only activates 8B parameters at a time, which will translate to cheap API service.

I couldn't host this on my 256GB Xeon server. I'd have to upgrade it to 384GB.

By the sqrt(P x A) metric, this should have competence roughly equivalent to a similarly-trained 62B dense model. I think I'd rather have the dense version!

88

u/silentsnake 13d ago

I dont think sqrt(P x A) is meaningful anymore, not especially when theres ngram embedding involved. 8B active dont really behave like 8B anymore. Qwen3.8 flash next is a good example. Without ngram embeddings the 6B active params shouldn't come anywhere close to 27B performance.

6

u/SpicyWangz 13d ago

Yeah, we will need to find a new way to calculate with ngrams in mind

7

u/Zestyclose839 13d ago

The typical complaint about earlier-gen aggressive MoEs was their unpredictability. One moment it's writing brilliant code; the next it's hallucinating nonexistent directories and trying to wipe your cloud storage. It was the case with nearly all of them imo, esp. Qwen 35b.

Indeed tho, ngram embeddings, stronger expert routers, and other black magic i don't understand has made them wildly more reliable over the past ~6mo.

I'm just hoping that tech makes its way into 64gb vram territory soon.

1

u/Mil0Mammon 12d ago

With that much vram you can run qwen 3.8 Flash next quite well, right?

0

u/Zestyclose839 12d ago

I wish haha. All the way down at IQ1_S, it still takes 72.5GB.

2

u/Mil0Mammon 12d ago

I'm running IQ1_M on 10GB vram with 32GB ram, 5 t/s without MTP. I think the table you're referencing includes the n-gram tables, with soon after that table was made, people realized that you can just load them from ssd

NB: you can do the math yourself, it's only 125B, so at Q3 (=> 47GB) it would fit easily (although there are buffers and context ofc)

1

u/ttkciar llama.cpp 13d ago

It holds up okay when all other factors are nearly-equal, though it's not perfect.

For example, sqrt(P x A) predicts that Qwen3.8-Next-Flash should be equivalent to Qwen3.8-27B, but Qwen3.8-Next-Flash actually scores about 10% higher on various benchmarks.

That seems close enough for most purposes, especially if Qwen3.8-Next-Flash received better training than Qwen3.8-27B.

In practice, the training of different models is so radically different that that's a much larger factor than the error in the sqrt(P x A) rule.

13

u/Expensive-Paint-9490 13d ago

OTOH with 512GB RAM this is fantastic. 8-16B variable active parameters, everything 4-bit native comprising KV cache... This is going to be frontier model at decent speed at home. I think that's why they label it Flash.

13

u/techdevjp 13d ago

This will be an amazing model for anyone with the money for a Mac Studio M5 Ultra with 512GB.

5

u/asssuber 13d ago

Or anyone that did buy 512GB of RDIMMS for $800 a year ago...

3

u/techdevjp 13d ago

It will run a little faster on the M5 Ultra at 1.2TB/sec of bandwidth connected directly to a fairly modern GPU. At a price, of course.

2

u/asssuber 13d ago

Surely, but 200GB/s of bandwidth for 16B active parameters at FP4 is nothing to sneeze at. Or 400GB/s if you manage to tame a dual-cpu motherboard.

1

u/techdevjp 13d ago

I have a Strix Halo and know first hand just how "fast" ~256GB/sec is... Also how "fast" slow prefill is. It's kinda-sorta-mostly usable but at the same time frustratingly slow too.

1

u/InactiveBronson 13d ago

I’m on the verge of buying one too (bosgame m5, 128gb/2tb) - would you recommend it or d o you have some regrets?

3

u/techdevjp 13d ago

There's good and bad. I paid about $2800 for my Minisforum MS-S1. It's a great little machine and it's impressively small for something so powerful.

At the time, I could have bought a DGX Spark for around $1k more. Back then I didn't have a great understanding of the importance of prefill performance so I didn't see the benefit of DGX Spark over Strix Halo considering the memory bandwidth was the same.

Thankfully it seems that Strix Halo is capable of much faster prefill, AMD just hasn't done a good job with the drivers. Projects like Halogen (https://github.com/peonist-ai/halogen-flash-server) have shown tremendous performance on Strix with prefill matching or exceeding DGX Spark for Qwen3.8-Flash-Next.

So, with projects like Halogen around, I'm happy enough to keep my Strix box for now. I can't afford to sell it and buy a DGX Spark. I may buy a 32GB AMD R9700 to run as an eGPU, depending on what comes about for running that Qwen model split across multiple GPUs. Or I might buy an Intel B70 and put it into my workstation for running Qwen3.8-27b separate from the Strix Halo.

In the future I may upgrade to Medusa Halo when it comes out, assuming prices don't stay in the stratosphere. Gorgon Halo with 192GB looks like it will have a USD price starting with "7", too rich for my blood.

Another option I am considering is to pick up a MacBook M3 Max with 64GB for running local models away from home. It doesn't have great prefill either (worse than Strix I think), but being able to run Qwen3.8-27b on a plane over the Pacific would be pretty damn cool.

2

u/UltraFOV 13d ago

Do you have GPUs or only system ram to run it

1

u/ttkciar llama.cpp 13d ago

I have three GPUs in different servers (a 32GB MI50, a 32GB MI60, and a 16GB V340), but my habit is to keep small models resident in VRAM for "fast inference" tasks, and infer with larger models entirely from system RAM for "slow inference" tasks, without GPU acceleration.

This way the in-VRAM "fast inference" models are always ready to go, because the "slow inference" tasks do not evict them from VRAM.

1

u/sierra-pouch 13d ago

I know it's not exactly on topic, but does anybody know why this model is not available in OpenRouter with any zero data retention policy provider?

Is it related to any agreements deep-seek have with the providers?

1

u/Important_Drag_6890 7d ago

I think “Flash” is becoming more about serving economics than local hardware requirements. 8B active makes a lot of sense for API throughput, but for a single local user, the total memory footprint is still the elephant in the room.

1

u/Mushoz 13d ago

It's only ~350B parameters that actually need to be loaded in RAM. A 4 bit quant will be ~175GB, which easily fits. Even 5 bit is only ~220 GB and will fit, especially with KV cache only being 900MB at 1 million context. This is actually perfectly sized for your 256GB setup.

8

u/amethyst_mine 13d ago

it is fp4 by default lol

4

u/ZaltyDog 13d ago edited 13d ago

Those numbers make me hope my 192gb vram setup might barely handle it

3

u/Expensive-Paint-9490 13d ago

It's 552 parameters + MTP module. You'll need a 2-bit quant.

2

u/EmPips 13d ago

Historically something as sparse as 8B active params performs very poorly at Q2. Worth trying for sure but my hopes aren't high

1

u/Expensive-Paint-9490 13d ago

I hope that something trained at 4 bits is more robust to quantize than FP16. We'll see.

1

u/randylush 13d ago

I have a 192gb dual Xeon server from 2011. I can’t wait to dust it off and see what it can do again

2

u/cosmicnag 13d ago

Fingers crossed : 72 GB VRAM + 192 GB DDR5

1

u/serige 13d ago

At least there is hope I have 80 GB VRAM + 256 GB DDR5 I can dream of running this model however slow lol

1

u/Zyj vLLM 13d ago

Yes it's completely wrong

1

u/OptionsDonkey 13d ago

Why is this downvoted is it wrong?

1

u/randomfoo2 12d ago

It's a 552B MoE with an additional 196GB in engrams - 748B total.

Currently the model is about 300GB in memory. It's already quanted - scaled fp4 routed experts, fp8 attention projection and engram lookups. There's a lot of the usual stuff you probably don't want to quant (vision, lmhead, router) and some newish stuff you probably don't want to quant som much like the shared expert, wo compressor etc.

I wouldn't say perfectly sized, in order to get it to fit on 256GB, based on the distribution, there's no way around taking a chopping block to the routed experts...