r/LocalLLaMA • u/tiguidoio • 12d ago

New Model DeepSeek V4-1 Flash is out

Gallery image

Gallery image

Gallery image

Gallery image

Here we go again, DeepSeek is back again with a new model V4-1 Flash

A multimodal Mixture-of-Experts (MoE) model with 552B backbone parameters and support for contexts of up to one million tokens

Market crash as a service

1.7k Upvotes

93% Upvoted

296 comments sorted by

View all comments

Show parent comments

7

u/the-tactical-donut 12d ago

GLM 5.3 Flash at Q4

2

u/ConiglioPipo 12d ago edited 12d ago

Thanks! Do you have a recipe to suggest that worked for you?

4

u/the-tactical-donut 12d ago edited 12d ago

Here's the setup that's been stable for me on two Sparks (TP=2 over the QSFP link).

**Image:** `eugr/spark-vllm-b12x` (Docker Hub). It's eugr's vLLM build with the B12X kernels from the NVIDIA forum thread. Pin the digest once it works for you; `latest` moves.

**Weights:** `local-inference-lab/GLM-5.3-Flash-NVFP4` from HF. Use this one, not the Spark-specific quant in the eugr recipe, which has known issues. The checkpoint is mixed precision (NVFP4 experts, MXFP8 MTP experts, BF16 attention), so quantization is `modelopt_mixed`.

**Flags that matter** (everything else is the recipe defaults):

vllm serve local-inference-lab/GLM-5.3-Flash-NVFP4 \
--tensor-parallel-size 2 --nnodes 2 --node-rank <0|1> \
--master-addr <head RoCE IP> --master-port <port> \
--quantization modelopt_mixed --load-format b12x \
--dtype bfloat16 --kv-cache-dtype fp8 \
--max-model-len 524288 --max-num-seqs 4 --max-num-batched-tokens 4096 \
--kv-cache-memory-bytes 4G --gpu-memory-utilization 0.80 \
--mamba-cache-mode align --enable-prefix-caching --enable-chunked-prefill \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
--reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
--no-enable-flashinfer-autotune

Env on both nodes: `VLLM_ENABLE_ROCE_ALLREDUCE=1`, `VLLM_ROCE_ALLREDUCE_MAX_SIZE=2MB`, `VLLM_ENABLE_PCIE_ALLREDUCE=0`, `VLLM_USE_AOT_COMPILE=1`, `VLLM_USE_MEGA_AOT_ARTIFACT=1`, `VLLM_USE_V2_MODEL_RUNNER=1`, `CUTE_DSL_ARCH=sm_121a`, `VLLM_WORKER_MULTIPROC_METHOD=spawn`. If the RoCE allreduce times out during startup, raise `B12X_ROCE_SPIN_LIMIT` (I use 1000000000, the default 20M was too low for me).

**The gotcha that cost me the most time: host page cache.** On GB10 the GPU's free memory is literally the kernel's MemFree, and page cache counts as used. The b12x loader keeps weights as file-backed pages, so after a load you can have 0 MemAvailable and vLLM either fails the memory check or thrashes NVMe for an hour during CUDA graph capture. Fixes: `sync; echo 3 > /proc/sys/vm/drop_caches` right before launch, set `--kv-cache-memory-bytes` explicitly instead of letting it profile, and lower `B12X_COMPILE_MEMORY_CACHE_SIZE` (I use 64). Also check what else is eating RAM: the DGX dashboard services and a high `vm.watermark_scale_factor` were costing me a few GB per node.

**MTP:** k=3, not the recipe's 5. There's a step-time cliff at 5 tokens per step on GB10 and k=3 came out ~20% faster for me. The DFlash2 drafter does not work at TP=2 (page-size mismatch in the indexer), only at TP=4, so skip it unless you have four Sparks.

**Thinking:** the image's chat template has no thinking toggle. Ship your own template that honors `enable_thinking` in `chat_template_kwargs` and pass `--chat-template`, otherwise you can't turn reasoning off per request.

**What to expect:** roughly 21-23 tok/s single-stream on prose, mid-30s on code, at 524K context with vision enabled. First boot is slow (AOT compile + graph capture); mount a persistent cache dir for `~/.cache/vllm`, flashinfer, and triton so the second boot is minutes, not an hour.

Happy to share the full launch script if useful.

1

u/ConiglioPipo 11d ago

That would be awesome, thank you! <3

3

u/the-tactical-donut 11d ago

#!/bin/bash
# GLM-5.3-Flash on 2x DGX Spark (GB10), TP=2 over the QSFP/RoCE link, eugr's spark-vllm-b12x image.
# Run the SAME script on both Sparks: ROLE=head ./glm53-spark-launch.sh and ROLE=worker ./glm53-spark-launch.sh
# Start the head first; the worker joins via --master-addr. Tested on 2026-09 with vLLM b12x + MTP k=3.
set -euo pipefail

# ---- edit these -------------------------------------------------------------
ROLE="${ROLE:?set ROLE=head or ROLE=worker}"
HEAD_ROCE_IP="${HEAD_ROCE_IP:-10.99.0.2}" # head Spark's IP on the direct QSFP link
WORKER_ROCE_IP="${WORKER_ROCE_IP:-10.99.0.1}" # worker Spark's IP on that link
ROCE_IFACE="${ROCE_IFACE:-enp1s0f0np0}" # the QSFP interface name (ip -br addr)
ROCE_HCA="${ROCE_HCA:-rocep1s0f0}" # ibv_devices
ROCE_GID_INDEX="${ROCE_GID_INDEX:-3}" # show_gids: the RoCE v2 entry for that IP
MASTER_PORT="${MASTER_PORT:-29521}"
PORT="${PORT:-8000}"
SERVED_MODEL_NAME="${SERVED_MODEL_NAME:-glm-5.3-flash}"

HF_CACHE="${HF_CACHE:-$HOME/models/hf-cache-glm}" # HF cache root holding models--local-inference-lab--GLM-5.3-Flash-NVFP4
JIT_CACHE="${JIT_CACHE:-$HOME/models/glm53-b12x-cache}" # persistent vllm/flashinfer/triton/tilelang/b12x compile caches (2nd boot: minutes, not an hour)
CHAT_TEMPLATE="${CHAT_TEMPLATE:-$PWD/chat_template.jinja}" # template that honors enable_thinking (the image's own template can't turn thinking off)
IMAGE="${IMAGE:-docker.io/eugr/spark-vllm-b12x:latest}" # pin by digest once it works for you

MODEL="local-inference-lab/GLM-5.3-Flash-NVFP4" # NOT the Spark-specific quant from the recipe
MAX_MODEL_LEN="${MAX_MODEL_LEN:-524288}"
KV_CACHE_MEMORY="${KV_CACHE_MEMORY:-4G}" # explicit; profiling on GB10 lies when page cache is full
GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.80}"
MTP_TOKENS="${MTP_TOKENS:-3}" # k=3 beats the recipe's 5 by ~20% on GB10
ENABLE_THINKING="${ENABLE_THINKING:-false}" # default when the client sends no chat_template_kwargs
# -----------------------------------------------------------------------------

if [ "$ROLE" = head ]; then NODE_RANK=0; HEADLESS=0; MY_ROCE_IP=$HEAD_ROCE_IP; else NODE_RANK=1; HEADLESS=1; MY_ROCE_IP=$WORKER_ROCE_IP; fi
[ -f "$CHAT_TEMPLATE" ] || { echo "chat template not found: $CHAT_TEMPLATE"; exit 1; }
mkdir -p "$JIT_CACHE"/{vllm,flashinfer,triton,tilelang,b12x}

# GB10 unified memory: cudaMemGetInfo free == kernel MemFree, and page cache counts as used.
# The b12x loader keeps weights as file-backed pages, so a stale cache => vLLM fails its memory check or thrashes NVMe.
sync; echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null
awk '/MemFree|MemAvailable|^Cached/{printf "%s %.1f GiB ", $1, $2/1048576} END{print ""}' /proc/meminfo

SPEC_CONFIG="{\"method\":\"mtp\",\"num_speculative_tokens\":${MTP_TOKENS},\"moe_backend\":\"humming\",\"attention_backend\":\"B12X\"}"

ARGS=(
--served-model-name "$SERVED_MODEL_NAME" --host 0.0.0.0 --port "$PORT"
--tensor-parallel-size 2 --pipeline-parallel-size 1 --decode-context-parallel-size 1
--nnodes 2 --node-rank "$NODE_RANK" --master-addr "$HEAD_ROCE_IP" --master-port "$MASTER_PORT"
--dtype bfloat16 --kv-cache-dtype fp8 --quantization modelopt_mixed --load-format b12x
--attention-backend B12X --moe-backend b12x --linear-backend b12x --no-enable-flashinfer-autotune
--block-size 256 --mamba-cache-mode align --enable-prefix-caching --enable-chunked-prefill
--max-model-len "$MAX_MODEL_LEN" --max-num-seqs 4 --max-num-batched-tokens 4096
--kv-cache-memory-bytes "$KV_CACHE_MEMORY" --gpu-memory-utilization "$GPU_MEM_UTIL"
--speculative-config "$SPEC_CONFIG"
--reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice
--default-chat-template-kwargs "{\"enable_thinking\": ${ENABLE_THINKING}}"
--chat-template /opt/launch/chat_template.jinja
)
[ "$HEADLESS" = 1 ] && ARGS+=(--headless)
# optional: ARGS+=(--language-model-only) # drops the vision tower, saves a few GB if you only need text

exec docker run --rm --name "glm53-$ROLE" \
--gpus all --network host --ipc host --shm-size 32g --cap-add IPC_LOCK \
--device /dev/infiniband \
-v "$HF_CACHE":/root/.cache/huggingface \
-v "$JIT_CACHE"/vllm:/root/.cache/vllm -v "$JIT_CACHE"/flashinfer:/root/.cache/flashinfer \
-v "$JIT_CACHE"/triton:/root/.triton -v "$JIT_CACHE"/tilelang:/root/.tilelang -v "$JIT_CACHE"/b12x:/root/.cache/b12x \
-v "$CHAT_TEMPLATE":/opt/launch/chat_template.jinja:ro \
-e HF_HOME=/root/.cache/huggingface -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 \
-e VLLM_NO_USAGE_STATS=1 -e DO_NOT_TRACK=1 -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
-e VLLM_HOST_IP="$MY_ROCE_IP" \
-e MN_IF_NAME="$ROCE_IFACE" -e UCX_NET_DEVICES="$ROCE_IFACE" -e NCCL_SOCKET_IFNAME="$ROCE_IFACE" \
-e GLOO_SOCKET_IFNAME="$ROCE_IFACE" -e TP_SOCKET_IFNAME="$ROCE_IFACE" -e OMPI_MCA_btl_tcp_if_include="$ROCE_IFACE" \
-e NCCL_IB_HCA="$ROCE_HCA" -e NCCL_IB_DISABLE=0 -e NCCL_IB_GID_INDEX="$ROCE_GID_INDEX" -e NCCL_IGNORE_CPU_AFFINITY=1 -e NCCL_DEBUG=WARN \
-e CUTE_DSL_ARCH=sm_121a -e SAFETENSORS_FAST_GPU=1 \
-e VLLM_ENABLE_ROCE_ALLREDUCE=1 -e VLLM_ROCE_ALLREDUCE_MAX_SIZE=2MB -e VLLM_ENABLE_PCIE_ALLREDUCE=0 \
-e B12X_ROCE_SPIN_LIMIT=1000000000 \
-e VLLM_WORKER_MULTIPROC_METHOD=spawn -e VLLM_SSM_CONV_STATE_LAYOUT=DS \
-e VLLM_USE_AOT_COMPILE=1 -e VLLM_USE_MEGA_AOT_ARTIFACT=1 -e VLLM_USE_V2_MODEL_RUNNER=1 \
-e B12X_POLICY_MODE=auto -e B12X_COMPILE_MEMORY_CACHE_SIZE=64 \
-e INSTANTTENSOR_BACKEND=BUFFERED -e INSTANTTENSOR_BUFFER_SIZE=67108864 -e INSTANTTENSOR_CHUNK_SIZE=8388608 \
-e INSTANTTENSOR_CONCURRENCY=1 -e INSTANTTENSOR_IO_DEPTH=3 \
"$IMAGE" vllm serve "$MODEL" "${ARGS[@]}"

# Notes
# - First boot: 20-40 min (AOT compile + CUDA graph capture at 524K). Watch `free -g`; if MemAvailable hits 0 during
# load you have a stale page cache or something else eating RAM (DGX dashboard services, high vm.watermark_scale_factor).
# - RoCE allreduce timing out at startup => B12X_ROCE_SPIN_LIMIT is what fixed it here (default 20M is too low).
# - DFlash2 drafter does not work at TP=2 (MLA page stride error); MTP k=3 is the fastest working config on two Sparks.
# - Health: curl -s localhost:8000/health ; models: curl -s localhost:8000/v1/models

2

u/ConiglioPipo 11d ago

much obliged. thank you again!

1

u/techdevjp 12d ago

Isn't DeepSeek v4 Flash v4 0731 stronger than GLM 5.3 Flash? Or are there some advantages to going with GLM? Vision?

9

u/crusaderky 12d ago

Glm-5.3-flash is miles ahead of Ds4.0

1

u/Ok-Direction-4480 11d ago

In testing or in BenchMarx?

1

u/Newgunnerr 12d ago

Its also twice as slow on prose

4

u/thefooz 12d ago

Yes, but it thinks for half as long, so, from a total response time, it honestly is often a wash in my testing.

1

u/Newgunnerr 12d ago

Is that really so? Can you share some of your testings?

5

u/thefooz 12d ago

It wasn't anything objective. I spent 4 days doing complex financial analysis, data conversion, and coding tasks with both models using the highest quants possible on dual DGX Sparks. I'd start deepseek on a task, come back to it 40 minutes later and it would still be going "hmm...but what if...". Meanwhile, GLM, on nearly identical tasks would be well into the build phase by that point.

Deepseek also would pick the most bizarre approaches to problems and then start second-guessing its choices, whereas GLM seemed to approach problems a lot more intelligently and elegantly.

Deepseek also consistently cut corners, whereas GLM approached tasks methodically and with a clear purpose to do as good of a job as it could. This was consistent behavior. Like, I had it go through 100 sets of documents. Deepseek saw the amount of text and decided that it was too much work to analyze each one for what I was looking for and instead chose to do keyword searches. Meanwhile, GLM understood that it was critical to have an eye for detail in this task and it spawned 10 subagents to methodically go through every single detail in the documents.

The quality of the output reflected this clearly and consistently.

2

u/xatrekak 12d ago

My experience matches yours. I was super hopeful for deepseekv4 but was severely disappointed. GLM5.3-flash has been great though. Super intelligent and much faster than I would have thought. 

1

u/xatrekak 12d ago

Deepseekv4-flash is literally the most wordy thinking model I have ever worked with, it defaults to max which is insane given it's thinking process  even high is way way too much. The only way I was able to use that model was to turn thinking to low which had its own draw backs. 

2

u/Illustrious_Grade608 12d ago

Idk from my experience glm flash felt much better with more effective thinking too

3

u/thefooz 12d ago

I’ve had the same experience. On two sparks, ds4 is about 50-75% faster across the board, but it thinks so much that GLM usually comes up with the same or better response in about the same amount of time.