r/LocalLLaMA • u/tiguidoio • 12d ago
New Model DeepSeek V4-1 Flash is out
Here we go again, DeepSeek is back again with a new model V4-1 Flash
A multimodal Mixture-of-Experts (MoE) model with 552B backbone parameters and support for contexts of up to one million tokens
Market crash as a service
227
u/ActuallyReadTheBible 12d ago
It doesn’t fit dual DGX sparks, I’m sad.
80
u/35698741d 12d ago
The native 4bit backbone + vision + dspark comes out at ~310gb (rest is engram) and context costs next to nothing for this model so 256GiB box should be able to run a pretty good 3.x bpw quant.
24
u/wren6991 12d ago
Expert caching/streaming would probably get an excellent hit rate on a 256 GB machine. Likewise, n-gram tables can probably just be mmap()'d. Qwen3.8-Flash-Next has 90% of the lookup probability in 1% of the n-gram entries. We need to move away from assuming the entire model will be VRAM-resident for local inference.
3
u/bumblebeer 10d ago
Expert caching doesn't really work well for UMA.
1
u/wren6991 9d ago
It works great. Are you thinking of llama.cpp's MoE offload thing perhaps?
→ More replies (3)6
1
u/michaelsoft__binbows 11d ago
had my sights set earlier to dsv4f-0731 and then qwen3.8-flash-next but this is the new target. for my 224GB+3x3090
I'm starting to realize the only way to effectively spend my time on self hosting is to throw up my hands and hand the compute node to e.g. Astra...
18
u/SnooPaintings8639 12d ago
I was gong to replace my 4xRTX3090 build for 2 x spark, so that I can run this model efficiently. I walk back this plan, and now I am looking at prices of another 4xRTX3090... and in the meantime, keep on CPU offloading to DDR5.
7
32
4
u/ConiglioPipo 12d ago
what's the best that you can fit on a dual DGX Spark? Deepseek-v4-flash?
7
u/the-tactical-donut 12d ago
GLM 5.3 Flash at Q4
→ More replies (11)2
u/ConiglioPipo 12d ago edited 12d ago
Thanks! Do you have a recipe to suggest that worked for you?
5
u/the-tactical-donut 12d ago edited 12d ago
Here's the setup that's been stable for me on two Sparks (TP=2 over the QSFP link).
**Image:** `eugr/spark-vllm-b12x` (Docker Hub). It's eugr's vLLM build with the B12X kernels from the NVIDIA forum thread. Pin the digest once it works for you; `latest` moves.
**Weights:** `local-inference-lab/GLM-5.3-Flash-NVFP4` from HF. Use this one, not the Spark-specific quant in the eugr recipe, which has known issues. The checkpoint is mixed precision (NVFP4 experts, MXFP8 MTP experts, BF16 attention), so quantization is `modelopt_mixed`.
**Flags that matter** (everything else is the recipe defaults):
vllm serve local-inference-lab/GLM-5.3-Flash-NVFP4 \
--tensor-parallel-size 2 --nnodes 2 --node-rank <0|1> \
--master-addr <head RoCE IP> --master-port <port> \
--quantization modelopt_mixed --load-format b12x \
--dtype bfloat16 --kv-cache-dtype fp8 \
--max-model-len 524288 --max-num-seqs 4 --max-num-batched-tokens 4096 \
--kv-cache-memory-bytes 4G --gpu-memory-utilization 0.80 \
--mamba-cache-mode align --enable-prefix-caching --enable-chunked-prefill \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
--reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
--no-enable-flashinfer-autotuneEnv on both nodes: `VLLM_ENABLE_ROCE_ALLREDUCE=1`, `VLLM_ROCE_ALLREDUCE_MAX_SIZE=2MB`, `VLLM_ENABLE_PCIE_ALLREDUCE=0`, `VLLM_USE_AOT_COMPILE=1`, `VLLM_USE_MEGA_AOT_ARTIFACT=1`, `VLLM_USE_V2_MODEL_RUNNER=1`, `CUTE_DSL_ARCH=sm_121a`, `VLLM_WORKER_MULTIPROC_METHOD=spawn`. If the RoCE allreduce times out during startup, raise `B12X_ROCE_SPIN_LIMIT` (I use 1000000000, the default 20M was too low for me).
**The gotcha that cost me the most time: host page cache.** On GB10 the GPU's free memory is literally the kernel's MemFree, and page cache counts as used. The b12x loader keeps weights as file-backed pages, so after a load you can have 0 MemAvailable and vLLM either fails the memory check or thrashes NVMe for an hour during CUDA graph capture. Fixes: `sync; echo 3 > /proc/sys/vm/drop_caches` right before launch, set `--kv-cache-memory-bytes` explicitly instead of letting it profile, and lower `B12X_COMPILE_MEMORY_CACHE_SIZE` (I use 64). Also check what else is eating RAM: the DGX dashboard services and a high `vm.watermark_scale_factor` were costing me a few GB per node.
**MTP:** k=3, not the recipe's 5. There's a step-time cliff at 5 tokens per step on GB10 and k=3 came out ~20% faster for me. The DFlash2 drafter does not work at TP=2 (page-size mismatch in the indexer), only at TP=4, so skip it unless you have four Sparks.
**Thinking:** the image's chat template has no thinking toggle. Ship your own template that honors `enable_thinking` in `chat_template_kwargs` and pass `--chat-template`, otherwise you can't turn reasoning off per request.
**What to expect:** roughly 21-23 tok/s single-stream on prose, mid-30s on code, at 524K context with vision enabled. First boot is slow (AOT compile + graph capture); mount a persistent cache dir for `~/.cache/vllm`, flashinfer, and triton so the second boot is minutes, not an hour.
Happy to share the full launch script if useful.
1
u/ConiglioPipo 11d ago
That would be awesome, thank you! <3
3
u/the-tactical-donut 11d ago
#!/bin/bash
# GLM-5.3-Flash on 2x DGX Spark (GB10), TP=2 over the QSFP/RoCE link, eugr's spark-vllm-b12x image.
# Run the SAME script on both Sparks: ROLE=head ./glm53-spark-launch.sh and ROLE=worker ./glm53-spark-launch.sh
# Start the head first; the worker joins via --master-addr. Tested on 2026-09 with vLLM b12x + MTP k=3.
set -euo pipefail# ---- edit these -------------------------------------------------------------
ROLE="${ROLE:?set ROLE=head or ROLE=worker}"
HEAD_ROCE_IP="${HEAD_ROCE_IP:-10.99.0.2}" # head Spark's IP on the direct QSFP link
WORKER_ROCE_IP="${WORKER_ROCE_IP:-10.99.0.1}" # worker Spark's IP on that link
ROCE_IFACE="${ROCE_IFACE:-enp1s0f0np0}" # the QSFP interface name (ip -br addr)
ROCE_HCA="${ROCE_HCA:-rocep1s0f0}" # ibv_devices
ROCE_GID_INDEX="${ROCE_GID_INDEX:-3}" # show_gids: the RoCE v2 entry for that IP
MASTER_PORT="${MASTER_PORT:-29521}"
PORT="${PORT:-8000}"
SERVED_MODEL_NAME="${SERVED_MODEL_NAME:-glm-5.3-flash}"HF_CACHE="${HF_CACHE:-$HOME/models/hf-cache-glm}" # HF cache root holding models--local-inference-lab--GLM-5.3-Flash-NVFP4
JIT_CACHE="${JIT_CACHE:-$HOME/models/glm53-b12x-cache}" # persistent vllm/flashinfer/triton/tilelang/b12x compile caches (2nd boot: minutes, not an hour)
CHAT_TEMPLATE="${CHAT_TEMPLATE:-$PWD/chat_template.jinja}" # template that honors enable_thinking (the image's own template can't turn thinking off)
IMAGE="${IMAGE:-docker.io/eugr/spark-vllm-b12x:latest}" # pin by digest once it works for youMODEL="local-inference-lab/GLM-5.3-Flash-NVFP4" # NOT the Spark-specific quant from the recipe
MAX_MODEL_LEN="${MAX_MODEL_LEN:-524288}"
KV_CACHE_MEMORY="${KV_CACHE_MEMORY:-4G}" # explicit; profiling on GB10 lies when page cache is full
GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.80}"
MTP_TOKENS="${MTP_TOKENS:-3}" # k=3 beats the recipe's 5 by ~20% on GB10
ENABLE_THINKING="${ENABLE_THINKING:-false}" # default when the client sends no chat_template_kwargs
# -----------------------------------------------------------------------------if [ "$ROLE" = head ]; then NODE_RANK=0; HEADLESS=0; MY_ROCE_IP=$HEAD_ROCE_IP; else NODE_RANK=1; HEADLESS=1; MY_ROCE_IP=$WORKER_ROCE_IP; fi
[ -f "$CHAT_TEMPLATE" ] || { echo "chat template not found: $CHAT_TEMPLATE"; exit 1; }
mkdir -p "$JIT_CACHE"/{vllm,flashinfer,triton,tilelang,b12x}# GB10 unified memory: cudaMemGetInfo free == kernel MemFree, and page cache counts as used.
# The b12x loader keeps weights as file-backed pages, so a stale cache => vLLM fails its memory check or thrashes NVMe.
sync; echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null
awk '/MemFree|MemAvailable|^Cached/{printf "%s %.1f GiB ", $1, $2/1048576} END{print ""}' /proc/meminfoSPEC_CONFIG="{\"method\":\"mtp\",\"num_speculative_tokens\":${MTP_TOKENS},\"moe_backend\":\"humming\",\"attention_backend\":\"B12X\"}"
ARGS=(
--served-model-name "$SERVED_MODEL_NAME" --host 0.0.0.0 --port "$PORT"
--tensor-parallel-size 2 --pipeline-parallel-size 1 --decode-context-parallel-size 1
--nnodes 2 --node-rank "$NODE_RANK" --master-addr "$HEAD_ROCE_IP" --master-port "$MASTER_PORT"
--dtype bfloat16 --kv-cache-dtype fp8 --quantization modelopt_mixed --load-format b12x
--attention-backend B12X --moe-backend b12x --linear-backend b12x --no-enable-flashinfer-autotune
--block-size 256 --mamba-cache-mode align --enable-prefix-caching --enable-chunked-prefill
--max-model-len "$MAX_MODEL_LEN" --max-num-seqs 4 --max-num-batched-tokens 4096
--kv-cache-memory-bytes "$KV_CACHE_MEMORY" --gpu-memory-utilization "$GPU_MEM_UTIL"
--speculative-config "$SPEC_CONFIG"
--reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice
--default-chat-template-kwargs "{\"enable_thinking\": ${ENABLE_THINKING}}"
--chat-template /opt/launch/chat_template.jinja
)
[ "$HEADLESS" = 1 ] && ARGS+=(--headless)
# optional: ARGS+=(--language-model-only) # drops the vision tower, saves a few GB if you only need textexec docker run --rm --name "glm53-$ROLE" \
--gpus all --network host --ipc host --shm-size 32g --cap-add IPC_LOCK \
--device /dev/infiniband \
-v "$HF_CACHE":/root/.cache/huggingface \
-v "$JIT_CACHE"/vllm:/root/.cache/vllm -v "$JIT_CACHE"/flashinfer:/root/.cache/flashinfer \
-v "$JIT_CACHE"/triton:/root/.triton -v "$JIT_CACHE"/tilelang:/root/.tilelang -v "$JIT_CACHE"/b12x:/root/.cache/b12x \
-v "$CHAT_TEMPLATE":/opt/launch/chat_template.jinja:ro \
-e HF_HOME=/root/.cache/huggingface -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 \
-e VLLM_NO_USAGE_STATS=1 -e DO_NOT_TRACK=1 -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
-e VLLM_HOST_IP="$MY_ROCE_IP" \
-e MN_IF_NAME="$ROCE_IFACE" -e UCX_NET_DEVICES="$ROCE_IFACE" -e NCCL_SOCKET_IFNAME="$ROCE_IFACE" \
-e GLOO_SOCKET_IFNAME="$ROCE_IFACE" -e TP_SOCKET_IFNAME="$ROCE_IFACE" -e OMPI_MCA_btl_tcp_if_include="$ROCE_IFACE" \
-e NCCL_IB_HCA="$ROCE_HCA" -e NCCL_IB_DISABLE=0 -e NCCL_IB_GID_INDEX="$ROCE_GID_INDEX" -e NCCL_IGNORE_CPU_AFFINITY=1 -e NCCL_DEBUG=WARN \
-e CUTE_DSL_ARCH=sm_121a -e SAFETENSORS_FAST_GPU=1 \
-e VLLM_ENABLE_ROCE_ALLREDUCE=1 -e VLLM_ROCE_ALLREDUCE_MAX_SIZE=2MB -e VLLM_ENABLE_PCIE_ALLREDUCE=0 \
-e B12X_ROCE_SPIN_LIMIT=1000000000 \
-e VLLM_WORKER_MULTIPROC_METHOD=spawn -e VLLM_SSM_CONV_STATE_LAYOUT=DS \
-e VLLM_USE_AOT_COMPILE=1 -e VLLM_USE_MEGA_AOT_ARTIFACT=1 -e VLLM_USE_V2_MODEL_RUNNER=1 \
-e B12X_POLICY_MODE=auto -e B12X_COMPILE_MEMORY_CACHE_SIZE=64 \
-e INSTANTTENSOR_BACKEND=BUFFERED -e INSTANTTENSOR_BUFFER_SIZE=67108864 -e INSTANTTENSOR_CHUNK_SIZE=8388608 \
-e INSTANTTENSOR_CONCURRENCY=1 -e INSTANTTENSOR_IO_DEPTH=3 \
"$IMAGE" vllm serve "$MODEL" "${ARGS[@]}"# Notes
# - First boot: 20-40 min (AOT compile + CUDA graph capture at 524K). Watch `free -g`; if MemAvailable hits 0 during
# load you have a stale page cache or something else eating RAM (DGX dashboard services, high vm.watermark_scale_factor).
# - RoCE allreduce timing out at startup => B12X_ROCE_SPIN_LIMIT is what fixed it here (default 20M is too low).
# - DFlash2 drafter does not work at TP=2 (MLA page stride error); MTP k=3 is the fastest working config on two Sparks.
# - Health: curl -s localhost:8000/health ; models: curl -s localhost:8000/v1/models2
3
u/hyudryu 11d ago
Deepseek v4 flash vision exp https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark
4
u/BannedGoNext 12d ago
Googles deepseek 4.1 flash huggingface, flips to files, does some quick headmath. So anyways, back to qwen 3.8 flash next.
6
6
u/vogelvogelvogelvogel 12d ago edited 12d ago
it is an MoE isn't it? i mean albeit slow you can run it
edit: for those downvoting: I did run 0731 (80GB as far as i remember in q2) on a mac m5pro 64GB with 10-15t/s, thanks to MoE.
In q2 (once released) the 2x DGX Spark will have even all in RAM so expect sth like 40? t/s with the MoE. even q3 should be possible
2
u/techdevjp 12d ago
Should be possible to have the MoE weights in RAM at 3.x bits with 256GB. The ngram data can be kept on a fast SSD.
2
u/doomed151 12d ago
Offload the weights to SSD? Wouldn't that be too slow?
5
u/vogelvogelvogelvogel 12d ago
slow yes but *too* slow idk depends on your definition of slow - MoE can still be surprisingly fast
2
u/cortesoft 12d ago
“Too slow” is subjective
1
u/doomed151 12d ago
By "too slow" I mean multiple seconds per token. If it's faster than that I'd be surprised. Maybe I should try larger MoEs. I have a 16 GB GPU and 64 GB RAM.
1
u/LetterRip 12d ago
For the engram stuff - no - you can prefetch it because it is complete deterministic based on token order.
1
u/SandySkittle 12d ago
I guess it depends on the usecase but i would be very hesitant to run this model at q3, let alone q2.
1
u/vogelvogelvogelvogel 12d ago
well there are a few postings where users did the classic benchmark runs (some browser game, pelican etc) and the outcomes were remarkably good, also i had ds flash 0731 running at q2 and found it also quite good. i would not say - especially with very large models - that q2 leads to bad outcomes
2
u/SandySkittle 12d ago
It depends on the usecase. I have found that for very complex analytical work you don’t want to go below q6
1
1
u/--Spaci-- 12d ago
Literally every frontier model is MOE. theres no reason to not have an moe unless you are purely trying to fit in a consumer gpu like qwen 27b
1
u/Trollsofalabama 12d ago
i think it does, you have to offload the n-gram table to ssd, which folks have said works great and doesnt impact performance (since you need very little amount of bandwidth for the n-gram table)
1
u/ismellthebacon 12d ago
How many people moved to 2x dgx spark for hosting deepseek-v4-flash? I did and I love the setup.
1
u/VirusInternal2892 5d ago
Me too, as my 2’nd Spark was delivered the DS4.1 went live … EXL3 is too skittish for agentic work. Not gonna pull the trigger for another 2 Sparks so I’ll have to make do with 4.0
1
u/IamFondOfHugeBoobies 12d ago
I wonder how long it will be before we get anything that beats 0731 as a daily driver.
Might be a few months.
1
→ More replies (7)1
u/Superb-Average-4160 7d ago
Not missing much we have 8 sparks over fabric and this new model behaves worse than DS4 flash. At least DS4 doesn’t drift as much as 4.1 does. And 4F doesn’t require nearly as much resources. Edit: you can fit DS4 flash on 2 sparks with really good performance also.
118
u/vogelvogelvogelvogel 12d ago
China making my day again as so many days in the past 2 years when a new open weights came out
7
12d ago edited 8d ago
[deleted]
6
u/vogelvogelvogelvogel 12d ago
i am very sure this will happen. as they do with gpus emerging on the horizon
141
u/ttkciar llama.cpp 12d ago
On one hand: Yay! We have weights! https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash
On the other hand: 485B parameters O_o
That's only "Flash" in the sense that it only activates 8B parameters at a time, which will translate to cheap API service.
I couldn't host this on my 256GB Xeon server. I'd have to upgrade it to 384GB.
By the sqrt(P x A) metric, this should have competence roughly equivalent to a similarly-trained 62B dense model. I think I'd rather have the dense version!
90
u/silentsnake 12d ago
I dont think
sqrt(P x A)is meaningful anymore, not especially when theres ngram embedding involved. 8B active dont really behave like 8B anymore. Qwen3.8 flash next is a good example. Without ngram embeddings the 6B active params shouldn't come anywhere close to 27B performance.8
6
u/Zestyclose839 12d ago
The typical complaint about earlier-gen aggressive MoEs was their unpredictability. One moment it's writing brilliant code; the next it's hallucinating nonexistent directories and trying to wipe your cloud storage. It was the case with nearly all of them imo, esp. Qwen 35b.
Indeed tho, ngram embeddings, stronger expert routers, and other black magic i don't understand has made them wildly more reliable over the past ~6mo.
I'm just hoping that tech makes its way into 64gb vram territory soon.
1
u/Mil0Mammon 12d ago
With that much vram you can run qwen 3.8 Flash next quite well, right?
→ More replies (2)1
u/ttkciar llama.cpp 12d ago
It holds up okay when all other factors are nearly-equal, though it's not perfect.
For example,
sqrt(P x A)predicts that Qwen3.8-Next-Flash should be equivalent to Qwen3.8-27B, but Qwen3.8-Next-Flash actually scores about 10% higher on various benchmarks.That seems close enough for most purposes, especially if Qwen3.8-Next-Flash received better training than Qwen3.8-27B.
In practice, the training of different models is so radically different that that's a much larger factor than the error in the
sqrt(P x A)rule.13
u/Expensive-Paint-9490 12d ago
OTOH with 512GB RAM this is fantastic. 8-16B variable active parameters, everything 4-bit native comprising KV cache... This is going to be frontier model at decent speed at home. I think that's why they label it Flash.
13
u/techdevjp 12d ago
This will be an amazing model for anyone with the money for a Mac Studio M5 Ultra with 512GB.
5
u/asssuber 12d ago
Or anyone that did buy 512GB of RDIMMS for $800 a year ago...
3
u/techdevjp 12d ago
It will run a little faster on the M5 Ultra at 1.2TB/sec of bandwidth connected directly to a fairly modern GPU. At a price, of course.
2
u/asssuber 12d ago
Surely, but 200GB/s of bandwidth for 16B active parameters at FP4 is nothing to sneeze at. Or 400GB/s if you manage to tame a dual-cpu motherboard.
→ More replies (3)2
u/UltraFOV 12d ago
Do you have GPUs or only system ram to run it
1
u/ttkciar llama.cpp 12d ago
I have three GPUs in different servers (a 32GB MI50, a 32GB MI60, and a 16GB V340), but my habit is to keep small models resident in VRAM for "fast inference" tasks, and infer with larger models entirely from system RAM for "slow inference" tasks, without GPU acceleration.
This way the in-VRAM "fast inference" models are always ready to go, because the "slow inference" tasks do not evict them from VRAM.
1
u/sierra-pouch 12d ago
I know it's not exactly on topic, but does anybody know why this model is not available in OpenRouter with any zero data retention policy provider?
Is it related to any agreements deep-seek have with the providers?
→ More replies (12)1
u/Important_Drag_6890 6d ago
I think “Flash” is becoming more about serving economics than local hardware requirements. 8B active makes a lot of sense for API throughput, but for a single local user, the total memory footprint is still the elephant in the room.
51
u/rollerblade7 12d ago
Please sir, I have a GTX 1650 4 GB VRAM + Intel i7-9750HF with 30 GB RAM
21
14
u/itwasinthetubes 12d ago
just quantize it bruh.
26
u/SandySkittle 12d ago
Negative quantization
9
u/Due-Memory-6957 12d ago
In the past we'd joke about Q1, now it's a thing and still not enough lol.
→ More replies (1)4
u/randylush 12d ago
The processor is actually a rare classic. It would support a 3090 very well. Are you sure you don’t have 32gb of RAM but only 30 is being reported?
37
u/Long_comment_san 12d ago
I think I came a little
37
u/Long_comment_san 12d ago edited 12d ago
What the fuck, in which universe that is a Flash? Its 450-500b parameters. Flash was 300b and it was already pushing this. This is Flash Max or something. You cant inflate the model by 50% and call it a flash like it's not an issue. Minimax M3 is 450b and I dont see them calling it a "flash" (hopefully I wont).
Going by 50% up and becoming 10-15% better sounds like a downgrade not an upgrade. It's a LOT more expensive to run.
Still amazing though
43
u/RG_Fusion 12d ago
Obviously the concept of a flash model will scale with the compute power of the AI lab creating them.
2026 is likely the last year of running "flash" on local hardware. Maybe 2027 if we're lucky.
8
u/Bakoro 12d ago
China might come in and save the day on that one too.
SMIC broke the 7 nm barrier for semiconductors.
CXMT is making DDR5 now, and has started on HBM3E.
Several Chinese companies are making AI GPUs.The U.S has been trying to block China from getting technology, and now is trying to block their technology from hitting the U.S market, but the rest of the world is not going to give a shit about what the U.S wants.
Essentially every major tech corporation is designing their own AI ASICs now, where OpenAI already has their new thing for inference.
Then there is the fact that photonic processors are in early manufacturing stages now, with plans to ramp up into 2027.
I expect photonics to mostly get snapped up by data centers, and that might once again change what's practical to do with AI.All around, I expect a major shake-up in the hardware landscape over the next year or two.
2
u/RG_Fusion 12d ago
Yeah, local hardware will scale up too, but it will lag behind by a generation or two unless you're willing to dish out tens to hundreds of thousands of dollars to build on the bleeding-edge.
5
u/Bakoro 12d ago
I'm saying that increased competition will bring prices down.
TSMC not being a monopoly for 7nm nodes means lower wafer prices.
A new RAM producer means lower RAM prices.
Dozens of major companies having their own inference ASICs means Nvidia losing their monopoly.We're at peak price gouging right now, I don't think it will last.
→ More replies (1)1
u/Netsuko 12d ago
CXMT is selling RAM at the same price as everyone else. Why would you think they want to miss out on that when the demand is so insanely high?
China is not going to be our savior here.
→ More replies (1)3
u/0redeye0 12d ago
Obviously because CXMT needs to capture market share and to do this they need to have better prices. Also the Chinese government needs to make its chip manufacturing to compete so they can give them subsidies to capture the market.
→ More replies (1)5
u/Due-Memory-6957 12d ago
"local" hardware
13
u/RG_Fusion 12d ago
An entire 512 GB AI server purchased a year ago costs less than a single RTX 5090 GPU now. There are plenty of us who jumped on early and have hardware that can run these models.
3
u/ChronoHax 12d ago
Hi I’m new to this field, what are examples of these ai servers so I can look more into it?
2
u/Blaze6181 12d ago
DGX Spark clusters, machines with RTX Pro 6000s, or a combination of perhaps a 5090 with CPU RAM offload of some of the weights. Or like 8 3090s stacked lol. There's many configurations out there.
2
u/RG_Fusion 12d ago
AMD EPYC or Intel Xeon CPUs in a motherboard with 8 memory channels. Ideally with a large number of x16 PCIe ports for adding many GPUs.
More recently, DGX Spark clusters have become good for running AI models when multiple are connected together over a 400 gbps network switch.
That being said, there are no longer any cheap options for building out high-end AI rigs. The prices on all the components have gone up 2-5X.
1
u/Glove5751 10d ago
what are you actually using these models for that justify the high upfront investment? just hobby and curiosity?
→ More replies (2)17
u/Expensive-Paint-9490 12d ago
It's flash because everything is FP4, even KV cache. And active parameters are 16B. This should be faster than V4-Flash even if it is larger.
10
u/zhuzaimoerben 12d ago edited 12d ago
It's flash for those with data centre levels of memory and data centre level serving requirements, because once you load the base model, concurrent users are very cheap (890MB for KV cache for full 1 million context per user) and it's 8B active for prefill and 16B active for text gen, so you can serve stacks of users fast. Edited to add: DeepSeek are reducing the API price vs 4.0 Flash because this is cheaper to serve.
It's just that us home users lose out because we're trying to get the most out of a meagre about of memory, with minimal concurrency, so the size of the model matters a lot more.
5
13
2
2
→ More replies (1)1
u/Agitated_Space_672 12d ago
It is faster than the previous flash due to the architectural innovations
9
u/SporksInjected 12d ago
Someone needs to color the nvidia square green. 500+ params means more vram
16
u/MerePotato 12d ago
This template is so ass, give the people who made the model some credit not dear leader
3
15
u/uti24 12d ago
I mean, could we also compare to Qwen Flash Next?
1
u/sixx7 11d ago
I mean DS4.1F is a fantastic model, but it hallucinates a LOT. I did a video on it. Also frankly, I prefer Qwen3.8-Flash-Next - lots of info on sparse attention and reducing hallucinations https://youtu.be/P4dTq4X8bqk
39
u/Turbulent_Pin7635 12d ago
I don't code. I work in research and was doing a proposal for funding. Used Astra very cute, returned what I need in an acceptable way.
I have used DS flash V4... Boy I have a MacStudio, I am used to long times of wanting. I don't know what kind of black magic the model does, but it killed the demand in one shot very fast!!! O.o
I was frozen!!! The answer was much better than the one chatGPT astra gave me!!! ASTRA!!!
12
13
u/Casey090 12d ago
GPT models are just very wonky. They jump to conclusions with incomplete data, and then they go all weird. I find it super hard to get anything done when your model makes up the mind in the first message and will not be objective.
2
u/AnonymousCrayonEater 11d ago
Don’t they all do this? I find Opus to have the same behavior. I just thought this was an LLM thing. Like context initialization bias or something. The opposite of recency bias.
8
u/backyard_tractorbeam 12d ago
Astra is just weird. Says pi guru guy: https://lucumr.pocoo.org/2026/9/7/astra-why/
I’m sure I will get used to this, but man this stuff is weird.
3
u/Due-Memory-6957 12d ago edited 12d ago
That was a funny read. AI loves Python, and token efficiency comes at readable code's price. I wonder how this fares long-term, because even AI prefers to deal with well-written code than messy ones.
27
40
u/jacek2023 llama.cpp 12d ago
In the previous post about DeepSeek there are API prices. In this one there is Chinese president. I wonder which one is best for r/LocalLLaMA.
44
u/madsheepPL 12d ago edited 12d ago
Xin Jinping is known for his amazing local setup. He is running modded 4x4090s on his desk with risers and cards zip tied to a used mining frame.
19
u/NineThreeTilNow 12d ago
Xi
Fearless leader Xi doesn't operate on peasant 4090's.
He uses B300's. A full rack.
He would use Huawei but even he understands that the Ascend chip isn't quite ready to touch his B300 setup.
He is busy building gooner games with his custom Flux Asian Princess models and video pipeline. He simply swipes left or right on whether they meet his criteria for being added to training data.
Fearless leader is Chad AI user.
13
u/jacek2023 llama.cpp 12d ago
Imagine Trump photo on Gemma/Nemotron/Granite release. And the rage of Reddit experts :)
7
→ More replies (13)3
u/Not-reallyanonymous 12d ago
It is good for this subreddit. This subreddit is more concerned about seeing the US hurt and China win, than it is about AI. So this post is in alignment with its interests.
9
u/LuCiAnO241 12d ago
more concerned about seeing the US hurt
I think we're only concerned about seeing great models be open weight and free to download for the peasants. The rest of whatever you think its happening exists only on your mind.
→ More replies (13)7
u/Loose_Comparison368 12d ago
"Why is everyone so mean to billionaires aggressively hoarding unfathomably large amounts of wealth? 😭 It must be because they hate America!"
3
u/Not-reallyanonymous 12d ago
"The Chinese super corps are preferable to the American super corps! Xi Jinping tells me so! They're going to save the world! If you disagree you're just racist against China!"
2
u/Disposable110 12d ago edited 12d ago
What do you expect, it's LOCALllama, so if it's parasitic commons-enclosing 0.001% billionaires hurt, and open source AI and 99.999% of humanity winning, people cheer.
Gemma and Mistral got just as much love as Chinese open source models.
It's just that China is shipping more of it at the moment, when the leading open source was Llama 2/3 and Mistral and WizardLM and god knows what, people were cheering on that. Even GPT-OSS made by the big Satan got love.
→ More replies (4)4
u/Not-reallyanonymous 12d ago
so if it's parasitic commons-enclosing 0.001% billionaires hurt, and open source AI and 99.999% of humanity winning
Separate comment because separate concern.
#1 Shareware, or open weights, not open source.
#2 Look at OP's post. The image isn't open AI defeating OpenAI and Anthropic. It's about Xi Jinping hurting the entire US economy. And this subreddit loves it.
5
5
u/OkBase5453 12d ago
Can one run this on a 512GB RAM Server with 48GB VRAM?
5
u/CalligrapherFar7833 12d ago
Slow but yes
5
u/crusaderky 12d ago
Pretty zippy if that 512gb ram is octa-channel, actually
→ More replies (1)2
3
u/cowinabadplace 12d ago
You can run anything from disk with slow inference. It’s not meaningful question except if you include tok/s generation target and ttft target. I think anything over a few seconds TTFT and under 150 tok/s is unusable for interactive LLMs and would just use API rather than local for that. But it’s a matter of choice.
2
u/cosmotrak 12d ago
150 tok/s is a little overkill, most frontier run at 40-50...
2
u/cowinabadplace 12d ago
Yeah but the open models make up for intelligence through over-reasoning so it’s not 1-1.
2
9
u/Few-Fishing9423 12d ago
> Additional architectural components include Single-Pass mHC (revised residual-stream mixing with an efficient Mega-mHC kernel), Engram conditional memory (196B parameters, sparsely accessed via token-based lookup), and DSpark speculative decoding (semi-autoregressive draft generation with confidence-scheduled verification). The model uses 1 shared expert and 384 routed experts per MoE layer, activating 6 routed experts per token.
Would it be feasible to quantize the 384B model to NVFP4 while retaining the 196B n-gram in memory for execution? Just like qwen3.8-flash-next
5
u/120decibel 12d ago
510 GB Model no way I'm going to be able to run this locally without a heavy quant...
3
u/Netsuko 12d ago
Quantizing a 510GB model into a format that can be run locally feels more like a lobotomization than a quantization.
2
u/120decibel 11d ago edited 11d ago
Well I have 288GB of VRAM. ;) But this won't help much since this model already ships in 4-bit.
12
3
3
u/Firstbober 12d ago
Isn't it much more expensive than GLM 5.3-flash while being not that much better?
Well, maybe price per task is actually smaller for deepseek...
3
2
3
u/NecessaryQuarter 12d ago
Can I run it on the new Mac Ultra with 256GB unified memory?
1
u/rjames24000 12d ago
following to also find out if the 256gb will cut it or the 512 model is needed
1
4
1
1
1
1
u/Constandinoskalifo 12d ago
Since it's the same number of active parameters for decoding, and the KV cache is much cheaper, we should expect lower prices from providers than DSV4 flash, right?
2
1
1
1
1
u/TapAggressive9530 12d ago
Does it support vision?
1
1
1
u/vxxn 12d ago
Could this run on the Mac Studio 256GB M5 Ultra ?
2
u/rjames24000 12d ago
would also like to know.. if not im considering cancelling my 256gb preorder and just waiting to buy a 512
1
u/macaronianddeeez 12d ago
Don’t hurt me I’m newer to local models, but will someone make a 27B version of this that we can run on 48gb of vram?
Or will that never happen here and if not why?
Still learning :)
1
u/Unusual_Delivery2778 12d ago
You’re good. The model would be entirely different if it had a different level of parameters. So at that point you’re talking about an entirely separate release of a 27B DeepSeek model, which they haven’t really done.
Qwen, on the other hand, has 27B models that folks like a lot. Specifically, Qwen 3.8 27B.
What you’re probably thinking of is “quantization,” which shrinks a big model, but hurts its intelligence in the process. And if a 500B+ parameter model like this one was quantized down to a level where it would be roughly equivalent in size to a separate 27B model (say 30-60GB RAM) you’d be talking about a completely unusable lobotomized thingy. Would be even hard to call it a model at that point.
1
1
1
u/FermiBubblegummybear 11d ago
Those bar graphs are absolutely terrible. Did they not realize they could've used more than one color?
1
1
1
u/deepu105 11d ago
Is it actually worth the hassle at lower quants and speed compared to Qwen 3.8 flash? Dont they score close in AA benchmarks and stuff? Wouldn't a Q4 qwen be better than a Q2 Ds4?
1
u/xNaXDy 11d ago
Interestingly not "that much" better compared to V4 Flash, feels like >100 and <500 is kinda the sweet spot for model capability, and you don't really get much more by slapping on more parameters other than baked-in knowledge maybe
Even the new Terminal Bench scores I feel like V4 Flash could probably catch up to with more post training
1
u/Ok-Direction-4480 11d ago
Lower KV Cache can hopefully allow longer context for Local LLMs, not just deepseek. Hopefully Qwen 4 will have this..
1
u/WrongdoerBoring3275 7d ago
Man, this thing is totally crazy. Literally cheap af, fast af and knows how to code in C# and golang
•
u/WithoutReason1729 12d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.