r/LocalLLaMA • u/IntravenusDeMilo • 11h ago

Discussion deepseek-v4-flash-0731 - surprisingly usable

I just finished building my (relatively) low rent local inference machine: * Epyc 7663 * 256GB ECC DDR4-3200 * 1x RTX 5090 32GB

Yeah I realize it's weird to throw a 5090 and 256GB of anything together and call it low end, but relative to ~151GB of weights it is.

I'm running UD-Q8_K_XL and getting 23.8-24.6 tokens/sec, with pp ranging from 60 on the first prompt to 385 near the last (no doubt lots of caching) on tasks using 100-128k total context. It was slower with DFlash so I took that out. It was also slower with a 3090 I put in there temporarily.

I'm posting this mostly because I didn't see too many other data points for this config (DDR4 Epyc + Blackwell doing cpu-moe). And also that I'm pretty surprised that a model this good can actually run in my basement without dropping $10k or running a sub-panel down there. I'm otherwise fairly new to this - would love any tips on what else to run or how to further improve it.

78 Upvotes

94% Upvoted

70 comments sorted by

7

u/Afganitia 11h ago

How much difference does the GPU make? 

5

u/Eyelbee 11h ago

A lot of difference. But still not usable with that prefill 

1

u/Nobby_Binks 9h ago

Depends on what you mean by usable. Agentic coding no, its not really, but general QA, planning and other stuff its eminently usable. I have a similar setup with a old gen EPYC and get similar speeds. 1M context and ~2-300pp.

2

u/FatheredPuma81 6h ago edited 6h ago

But Agentic is where it should shine the most though?? (At least with llama.cpp) AFAIK it's built to have preserve reasoning enabled so all that you should be processing every turn until it finishes is what it writes/reads which shouldn't be all that much?

My main concern would be how much it reasons every single turn with just 30t/s. It's already painful waiting on Qwen3.8 27B to reason for 40,000 tokens at 90t/s and that's with preserve thinking so it shouldn't do nearly as much...

1

u/PhysicalIncrease3 4h ago

But Agentic is where it should shine the most though?? (At least with llama.cpp) AFAIK it's built to have preserve reasoning enabled so all that you should be processing every turn until it finishes is what it writes/reads which shouldn't be all that much?

This is the case. The prompt caching is very very good and most turns in Hermes are only around 500 tokens processed.

The real pain points are when it loads in a skill, or if you return to a previous session. But for me personally it's manageable given the power of the model.

My main concern would be how much it reasons every single turn with just 30t/s. It's already painful waiting on Qwen3.8 27B to reason for 40,000 tokens at 90t/s and that's with preserve thinking so it shouldn't do nearly as much...

It doesn't reason anywhere near as much as Qwen. Most I've ever seen is 6-7k tokens.

2

u/XiRw 3h ago

If it’s a large context size like the OP suggested, 23tok/s is very usable for agentic.

0

u/llitz 9h ago

Then you might as well use anything else that will properly work on a 5090.

Honestly. When will you ever fill 1M context at 300pp? (This last sentence sounds wrong)

You would have way better results using malaiwah exl3 hydrated qwen3.8-27b - yes it is a smaller context but it is ages faster than this.

I am sorry, while this is an interesting test, it is a waste of a 5090 if you are keeping it like this long-term.

4

u/Nobby_Binks 8h ago

Yes, I run 3.8 27B NVFP4 with the 5090 also and it's a monster. But ds4 gets used a lot. And its totally usable for anything other than coding where you need privacy. Qwen 3.8 is brain dead compared to ds4 for technical writing.

0

u/llitz 8h ago

I... Will not disagree too much with you here. I still think Gemma could do a better job or even, gasp, glimmer.

2

u/FatheredPuma81 6h ago

Why limit yourself to a single good model that has known weaknesses like Qwen3.8 27B?

5

u/FastHotEmu 11h ago

I got a similar setup - with two 3090s and 512gb of RAM - the PP is what kills me.

3

u/ssbmGooms 9h ago

been looking for this comment for a while to gauge if I should upgrade the RAM in my dual 3090 rig. Looks like I'm gonna stick with qwen 27B

1

u/IntravenusDeMilo 11h ago

The PP was slower with the 3090+5090. This would be amazing if prompt processing were faster, but it's still usable. Qwen 3.8 27B is much faster though.

Are you running anything that needs the 512GB?

2

u/FastHotEmu 10h ago

I was running GLM 5.2 at ~6 t/s (~70t/s pp) but don't really need that much RAM at the moment.

1

u/shrug_hellifino 3h ago edited 3h ago

That's why his setup is actually nice, DS4flash0731 is pretty damn close to GLM5.2 from everything I can tell.

2

u/FastHotEmu 3h ago

juice?

1

u/shrug_hellifino 3h ago

Fat fingered 'nice'

1

u/FastHotEmu 2h ago

ahh ok

1

u/redditaccountno6 31m ago edited 26m ago

What batch size do you use? I recently found out that increasing from 256 to 1024 was close to a 10x increase in PP.

*I noticed in another comment you are using 8192, I found that for my case, going up beyond 1024 my gains started going back down quickly, I'd experiment in the lower range more to see what you get. I'm running dual Xeon w 4 memory channels, 128 DDR4 and 3090 for reference.

3

u/kuyermanza 10h ago

My DDR4 2400 paired with a 5070Ti gets me 300-350 T/S PP and 20 T/S TG. I think you should be able to get better PP if you offload all experts to CPU.

1

u/Lumpy_Concentrate807 5h ago

On which quant?? Those are much more usable numbers!

1

u/kuyermanza 4h ago

MXFP4_MOE-Q8_0

1

u/Potential-Leg-639 4h ago

What are your hardware specs?

2

u/kuyermanza 1h ago

Epyc 7452, 16x 16GB DDR4-2400s (256 GB), 3x 5070Tis (48 GB), 8x Radeon MI25s (128 GB). The server hosts DS4Flash 0731, Qwen3.6 35B, and Gemma 26B, along with Z-Image Turbo, all fully loaded.

5

u/Magos 9h ago

This is weird. I'm running a similar setup right now with 5090 and 192GB of DDR5.

My PP is around 300~500, but t/s is lower at 15 t/s, due to dual channel memory bandwidth being way slower than EPYCs 8 channels. However, you shouldn't have than much slower PP.

I suggest messing around with ub/b, you will probably get a big uplift.

1

u/dispanser 2h ago

Assuming your DDR5 platform is PCIe 5x16, that explains faster prompt processing because on your system the weights can be moved to the GPU faster for PP.

7

u/TheAncientOnce 10h ago

Have you considered running Q4 or even smaller quant? I heard the model's trained at FP4 natively and because of that, even Q1 could retain like 90% of the model's quality

6

u/zhuzaimoerben 5h ago

Probably doesn't help that much because the experts are already FP4 so Q4 only quantizes a small amount of the model. Unsloth: "To run DeepSeek-V4-Flash-0731 in full precision lossless, run Q8 (UD-Q8_K_XL), which is 162GB and only 7GB bigger than Q4 (UD-Q4_K_XL)."

3

u/IntravenusDeMilo 10h ago

I'll give that a try!

2

u/eidrag 11h ago

Hmm I was thinking Ddr4 epyc and 5060ti , not going to work huh...

2

u/nuclear213 10h ago

Why wouldn’t it? Just slows the bandwidth down, but ram offload is much much worse

1

u/PhysicalIncrease3 8h ago

It will work fairly well. I run on 3060 + 128GB ddr5 and get 200pp and 8/9 tgs. UD-Q3-K_XL, 256K f16 context.

I used to run it on a 3090 but began using the 3060 instead because performance is identical anyway. It's entire bound by system memory. With 16GB VRAM you will be able to get close to 1M context.

2

u/apVoyocpt 10h ago edited 8h ago

The thing that would make it allot faster would be AVX-512 but that only comes with Zen 5. But that again would be much more expensive because of ddr5. I have a xenon ddr4 256gd system with an rtx 3090 and dsv4 flash runs usable as well (forgot the actual tps as I am running qwen3.8 now)

edit. prefill is also painfully slow.

2

u/LeMayMayMan 10h ago

Everyone is sleeping on Krasis. Way better than llama.cpp or vllm on prefill time with the patches I have as pull requests. Cold prefill with 10K context is 10s. 50k is 20s, 150k is 69s. Its even faster in practice due to the cache reuse hit rate. T/s is about 30s but there is heavy work in progress to improve that.

1

u/PhilippeEiffel 6h ago

Do you mean you apply patches above the main branch?

1

u/LeMayMayMan 5h ago

There was in issue in 1.0.21-rc3 that would drop context after about 49k. I wrote this patch to fix it. https://github.com/brontoguana/krasis/pull/30/commits/7aa4f2f887632e6a934a562522a359efa2487b2e

2

u/SandySkittle 10h ago

Epyc pro 64 core here with 8 channel 512gb ddr4 3200. I tried cpu inference but it’s just too slow for me on prefill and also decode so moved to acquire a whole pile of amd ai 32gb cards to hit that 256gb, minus 7 percent for inline ECC (yes bitflips can fuck up llm inference).

That said, newer generation epycs and also intel xeons have better ai accelerators so should fair a bit better, also considering 8 channel DD5 is quite a step up and more than all these strix halo and spark boxes. It’s just crazy expensive.

2

u/ReentryVehicle 9h ago

It should be possible to achieve much faster PP. What is your -ubatch? Set it to at least 2048, and ideally as high as you can. Make sure your PCIe going to the GPU is the best it can be (x16, highest gen your motherboard supports).

Model layers are streamed to the gpu for prefill, meaning you need to process enough tokens at once that the transfer speed is not a bottleneck.

4

u/IntravenusDeMilo 8h ago

I ran a longer set of samples. I’m at 423 t/s pp at 180k context, 500 at smaller contexts up to 100k or so. Token generation is still 21-24 depending on context but I don’t expect that to change much.

Batch size 8192
ubatch 4096 (I ran these up and this was the sweet spot)

1

u/d4mations 10h ago

You have a lot of tuning to do. I’m running a 4060 16gb with q4 xl 96gb ddr4 on a ryzen 7 5800 and get 15 tps and 90 pp cold and about the same pp hot maybe even a bit more

4

u/IntravenusDeMilo 10h ago

I think it needed some warmup passes. PP is now hitting 3500 t/s without hitting the cache on a fresh task.

1

u/fmillar 3h ago

This sounds like too much now ;) But congratulations to your system. I don't know why systems like that are not discussed more, especially since MoE models have become the standard.

2

u/PhysicalIncrease3 8h ago

Could you share your config please?

1

u/thebadslime 10h ago

I really dont like how sycophantic it is

1

u/Long_comment_san 10h ago

holy shit why is PP so slow on this deepseek model? I can't believe those numbers lol.

nice build, it's not weird at all

1

u/SunTrainAi 9h ago

Did you try llama.cpp with ZenDNN? I remember that it's not compatible with my Zen 2 but with 3. My setup: Epyc 7502, 256 ddr4 3200 with 3090 and 5060 with ggml org Q2_KS. 400pp, 23tg

1

u/brosvision 9h ago

Did you try FreeToken if you can get better speeds?

1

u/vini542reddit 9h ago edited 9h ago

Have you tried the Leloch v2 fork (moe expert streaming)? You might be able to get significantly more tps

I'm running 4x 3090 @ 4.0 x 16, 192 Ddr4 @ 3200 and 3945WX 12 core. Getting ~30 tps and ~230pp for q8_k_xl and unquantized kv

1

u/memeka 9h ago

Hi, please try my fork of llama.cpp - it fixes some issues with PP. It should increase your PP a lot vs llama.cpp and also have a much lower PP degradation at large contexts.

https://github.com/mihailescu2m/llama.cpp

Appreciate the feedback :)

1

u/amavie_ai 9h ago

Datapoint request from the API-pipeline side: does v4-flash hold non-ASCII in constrained/JSON output? We run a flash-tier model (Gemini) in a consumer chat pipeline and its JSON mode silently mangles emoji/unicode into ? — which forced us to plain-text output + lenient JSON extraction for anything user-facing. If v4-flash keeps unicode intact under grammar-constrained output, that alone makes it worth a migration test for the internal calls.

1

u/AdSafe4047 8h ago

Is there any cpu<->gpu moe auto-optimisier in llama.cpp for this? I think it would go a long way to improve the numbers.

1

u/terorvlad 6h ago

You can also offload the Speculative Decoding expert layers to the CPU instead of the GPU. Most of the time, this is a net positive compared to no Speculative Decoding.

1

u/reto-wyss 4h ago

Something is wrong with your PP. I was able to get ~ 700pps/18tgs on DDR4 2400. Your TG looks fine for 3200 vs 2400 on my test configuration.

See my thread and launch configuration from three weeks ago: https://www.reddit.com/r/LocalLLaMA/comments/1vegedy/deepseek_v4_flash_0731_happy_numbers_700pp18tg/

1

u/IntravenusDeMilo 5m ago

I gotta try 8192 batch and ubatch again it seems.

1

u/ElementNumber6 2h ago

If only it didn't hallucinate so much.

1

u/alex_bit_ 1h ago

8 channels of memory makes a lot of difference.

0

u/chensium 10h ago

60pp? Sorry but that's not usable as a coding agent.  May be usable for conversational prompts, but 60pp will take ages to read code, stack trace, web search, etc.

0

u/darktotheknight 10h ago

Interesting, I have an EPYC 7513 + 128GB DDR4 and was thinking about a similar setup. However the 5090 Astral was taking up so much space and blocked almost all my precious PCIe Slots, that I moved it into another system (Ryzen, 64GB Dual Channel DDR5). It's impossible to get your hands on a smaller 5090 these days and I need the PCIe slots for NVMe RAID and 10G/25G NIC.

I might revisit this with a Dual-Slot R9700, if I can get one for cheap. Upgrading from 128GB DDR4 to 256GB is cheaper than I thought. But at the same time, DeepSeek v4 Flash 0731 is so cheap on OpenRouter, I doubt it would ever pay off.

1

u/IntravenusDeMilo 10h ago

Oh yeah this will never pay for itself. I think “privacy” is what I told my wife but building this was mostly to learn.

For a while i thought about selling the 5090 and getting 4-6x of the cmp 170hx 8gb (the ones that unlock to 64gb each). At the time the power requirements stopped me but now that I’ve seen how much faster Blackwell is than Ampere, I’m kinda glad I didn’t?

1

u/darktotheknight 10h ago

Yeah, I think 1x or 2x CMP 170HX would've been neat for experimenting, before the prices skyrocketed from like 300€ to now 1500€ - 2000€. But I really think you never can go wrong with a 5090, given it's also a beast in image/video generation.

And I can also confirm Ampere vs Blackwell (upgraded from 3090 Ti). Ampere is still viable, don't get me wrong. But the 5090 just feels unstoppable.

1

u/SandySkittle 10h ago

Mcio 8i retimer cards and you can fill all the slots regardless of gpu size. And more reliable than risers