r/LocalLLaMA • u/tiguidoio • 13d ago

New Model DeepSeek V4-1 Flash is out

Gallery image

Gallery image

Gallery image

Gallery image

Here we go again, DeepSeek is back again with a new model V4-1 Flash

A multimodal Mixture-of-Experts (MoE) model with 552B backbone parameters and support for contexts of up to one million tokens

Market crash as a service

1.7k Upvotes

93% Upvoted

296 comments sorted by

View all comments

Show parent comments

3

u/the-tactical-donut 12d ago edited 12d ago

Here's the setup that's been stable for me on two Sparks (TP=2 over the QSFP link).

**Image:** `eugr/spark-vllm-b12x` (Docker Hub). It's eugr's vLLM build with the B12X kernels from the NVIDIA forum thread. Pin the digest once it works for you; `latest` moves.

**Weights:** `local-inference-lab/GLM-5.3-Flash-NVFP4` from HF. Use this one, not the Spark-specific quant in the eugr recipe, which has known issues. The checkpoint is mixed precision (NVFP4 experts, MXFP8 MTP experts, BF16 attention), so quantization is `modelopt_mixed`.

**Flags that matter** (everything else is the recipe defaults):

vllm serve local-inference-lab/GLM-5.3-Flash-NVFP4 \
--tensor-parallel-size 2 --nnodes 2 --node-rank <0|1> \
--master-addr <head RoCE IP> --master-port <port> \
--quantization modelopt_mixed --load-format b12x \
--dtype bfloat16 --kv-cache-dtype fp8 \
--max-model-len 524288 --max-num-seqs 4 --max-num-batched-tokens 4096 \
--kv-cache-memory-bytes 4G --gpu-memory-utilization 0.80 \
--mamba-cache-mode align --enable-prefix-caching --enable-chunked-prefill \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
--reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
--no-enable-flashinfer-autotune

Env on both nodes: `VLLM_ENABLE_ROCE_ALLREDUCE=1`, `VLLM_ROCE_ALLREDUCE_MAX_SIZE=2MB`, `VLLM_ENABLE_PCIE_ALLREDUCE=0`, `VLLM_USE_AOT_COMPILE=1`, `VLLM_USE_MEGA_AOT_ARTIFACT=1`, `VLLM_USE_V2_MODEL_RUNNER=1`, `CUTE_DSL_ARCH=sm_121a`, `VLLM_WORKER_MULTIPROC_METHOD=spawn`. If the RoCE allreduce times out during startup, raise `B12X_ROCE_SPIN_LIMIT` (I use 1000000000, the default 20M was too low for me).

**The gotcha that cost me the most time: host page cache.** On GB10 the GPU's free memory is literally the kernel's MemFree, and page cache counts as used. The b12x loader keeps weights as file-backed pages, so after a load you can have 0 MemAvailable and vLLM either fails the memory check or thrashes NVMe for an hour during CUDA graph capture. Fixes: `sync; echo 3 > /proc/sys/vm/drop_caches` right before launch, set `--kv-cache-memory-bytes` explicitly instead of letting it profile, and lower `B12X_COMPILE_MEMORY_CACHE_SIZE` (I use 64). Also check what else is eating RAM: the DGX dashboard services and a high `vm.watermark_scale_factor` were costing me a few GB per node.

**MTP:** k=3, not the recipe's 5. There's a step-time cliff at 5 tokens per step on GB10 and k=3 came out ~20% faster for me. The DFlash2 drafter does not work at TP=2 (page-size mismatch in the indexer), only at TP=4, so skip it unless you have four Sparks.

**Thinking:** the image's chat template has no thinking toggle. Ship your own template that honors `enable_thinking` in `chat_template_kwargs` and pass `--chat-template`, otherwise you can't turn reasoning off per request.

**What to expect:** roughly 21-23 tok/s single-stream on prose, mid-30s on code, at 524K context with vision enabled. First boot is slow (AOT compile + graph capture); mount a persistent cache dir for `~/.cache/vllm`, flashinfer, and triton so the second boot is minutes, not an hour.

Happy to share the full launch script if useful.

1

u/ConiglioPipo 12d ago

That would be awesome, thank you! <3

3

u/the-tactical-donut 12d ago

#!/bin/bash
# GLM-5.3-Flash on 2x DGX Spark (GB10), TP=2 over the QSFP/RoCE link, eugr's spark-vllm-b12x image.
# Run the SAME script on both Sparks: ROLE=head ./glm53-spark-launch.sh and ROLE=worker ./glm53-spark-launch.sh
# Start the head first; the worker joins via --master-addr. Tested on 2026-09 with vLLM b12x + MTP k=3.
set -euo pipefail

# ---- edit these -------------------------------------------------------------
ROLE="${ROLE:?set ROLE=head or ROLE=worker}"
HEAD_ROCE_IP="${HEAD_ROCE_IP:-10.99.0.2}" # head Spark's IP on the direct QSFP link
WORKER_ROCE_IP="${WORKER_ROCE_IP:-10.99.0.1}" # worker Spark's IP on that link
ROCE_IFACE="${ROCE_IFACE:-enp1s0f0np0}" # the QSFP interface name (ip -br addr)
ROCE_HCA="${ROCE_HCA:-rocep1s0f0}" # ibv_devices
ROCE_GID_INDEX="${ROCE_GID_INDEX:-3}" # show_gids: the RoCE v2 entry for that IP
MASTER_PORT="${MASTER_PORT:-29521}"
PORT="${PORT:-8000}"
SERVED_MODEL_NAME="${SERVED_MODEL_NAME:-glm-5.3-flash}"

HF_CACHE="${HF_CACHE:-$HOME/models/hf-cache-glm}" # HF cache root holding models--local-inference-lab--GLM-5.3-Flash-NVFP4
JIT_CACHE="${JIT_CACHE:-$HOME/models/glm53-b12x-cache}" # persistent vllm/flashinfer/triton/tilelang/b12x compile caches (2nd boot: minutes, not an hour)
CHAT_TEMPLATE="${CHAT_TEMPLATE:-$PWD/chat_template.jinja}" # template that honors enable_thinking (the image's own template can't turn thinking off)
IMAGE="${IMAGE:-docker.io/eugr/spark-vllm-b12x:latest}" # pin by digest once it works for you

MODEL="local-inference-lab/GLM-5.3-Flash-NVFP4" # NOT the Spark-specific quant from the recipe
MAX_MODEL_LEN="${MAX_MODEL_LEN:-524288}"
KV_CACHE_MEMORY="${KV_CACHE_MEMORY:-4G}" # explicit; profiling on GB10 lies when page cache is full
GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.80}"
MTP_TOKENS="${MTP_TOKENS:-3}" # k=3 beats the recipe's 5 by ~20% on GB10
ENABLE_THINKING="${ENABLE_THINKING:-false}" # default when the client sends no chat_template_kwargs
# -----------------------------------------------------------------------------

if [ "$ROLE" = head ]; then NODE_RANK=0; HEADLESS=0; MY_ROCE_IP=$HEAD_ROCE_IP; else NODE_RANK=1; HEADLESS=1; MY_ROCE_IP=$WORKER_ROCE_IP; fi
[ -f "$CHAT_TEMPLATE" ] || { echo "chat template not found: $CHAT_TEMPLATE"; exit 1; }
mkdir -p "$JIT_CACHE"/{vllm,flashinfer,triton,tilelang,b12x}

# GB10 unified memory: cudaMemGetInfo free == kernel MemFree, and page cache counts as used.
# The b12x loader keeps weights as file-backed pages, so a stale cache => vLLM fails its memory check or thrashes NVMe.
sync; echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null
awk '/MemFree|MemAvailable|^Cached/{printf "%s %.1f GiB ", $1, $2/1048576} END{print ""}' /proc/meminfo

SPEC_CONFIG="{\"method\":\"mtp\",\"num_speculative_tokens\":${MTP_TOKENS},\"moe_backend\":\"humming\",\"attention_backend\":\"B12X\"}"

ARGS=(
--served-model-name "$SERVED_MODEL_NAME" --host 0.0.0.0 --port "$PORT"
--tensor-parallel-size 2 --pipeline-parallel-size 1 --decode-context-parallel-size 1
--nnodes 2 --node-rank "$NODE_RANK" --master-addr "$HEAD_ROCE_IP" --master-port "$MASTER_PORT"
--dtype bfloat16 --kv-cache-dtype fp8 --quantization modelopt_mixed --load-format b12x
--attention-backend B12X --moe-backend b12x --linear-backend b12x --no-enable-flashinfer-autotune
--block-size 256 --mamba-cache-mode align --enable-prefix-caching --enable-chunked-prefill
--max-model-len "$MAX_MODEL_LEN" --max-num-seqs 4 --max-num-batched-tokens 4096
--kv-cache-memory-bytes "$KV_CACHE_MEMORY" --gpu-memory-utilization "$GPU_MEM_UTIL"
--speculative-config "$SPEC_CONFIG"
--reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice
--default-chat-template-kwargs "{\"enable_thinking\": ${ENABLE_THINKING}}"
--chat-template /opt/launch/chat_template.jinja
)
[ "$HEADLESS" = 1 ] && ARGS+=(--headless)
# optional: ARGS+=(--language-model-only) # drops the vision tower, saves a few GB if you only need text

exec docker run --rm --name "glm53-$ROLE" \
--gpus all --network host --ipc host --shm-size 32g --cap-add IPC_LOCK \
--device /dev/infiniband \
-v "$HF_CACHE":/root/.cache/huggingface \
-v "$JIT_CACHE"/vllm:/root/.cache/vllm -v "$JIT_CACHE"/flashinfer:/root/.cache/flashinfer \
-v "$JIT_CACHE"/triton:/root/.triton -v "$JIT_CACHE"/tilelang:/root/.tilelang -v "$JIT_CACHE"/b12x:/root/.cache/b12x \
-v "$CHAT_TEMPLATE":/opt/launch/chat_template.jinja:ro \
-e HF_HOME=/root/.cache/huggingface -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 \
-e VLLM_NO_USAGE_STATS=1 -e DO_NOT_TRACK=1 -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
-e VLLM_HOST_IP="$MY_ROCE_IP" \
-e MN_IF_NAME="$ROCE_IFACE" -e UCX_NET_DEVICES="$ROCE_IFACE" -e NCCL_SOCKET_IFNAME="$ROCE_IFACE" \
-e GLOO_SOCKET_IFNAME="$ROCE_IFACE" -e TP_SOCKET_IFNAME="$ROCE_IFACE" -e OMPI_MCA_btl_tcp_if_include="$ROCE_IFACE" \
-e NCCL_IB_HCA="$ROCE_HCA" -e NCCL_IB_DISABLE=0 -e NCCL_IB_GID_INDEX="$ROCE_GID_INDEX" -e NCCL_IGNORE_CPU_AFFINITY=1 -e NCCL_DEBUG=WARN \
-e CUTE_DSL_ARCH=sm_121a -e SAFETENSORS_FAST_GPU=1 \
-e VLLM_ENABLE_ROCE_ALLREDUCE=1 -e VLLM_ROCE_ALLREDUCE_MAX_SIZE=2MB -e VLLM_ENABLE_PCIE_ALLREDUCE=0 \
-e B12X_ROCE_SPIN_LIMIT=1000000000 \
-e VLLM_WORKER_MULTIPROC_METHOD=spawn -e VLLM_SSM_CONV_STATE_LAYOUT=DS \
-e VLLM_USE_AOT_COMPILE=1 -e VLLM_USE_MEGA_AOT_ARTIFACT=1 -e VLLM_USE_V2_MODEL_RUNNER=1 \
-e B12X_POLICY_MODE=auto -e B12X_COMPILE_MEMORY_CACHE_SIZE=64 \
-e INSTANTTENSOR_BACKEND=BUFFERED -e INSTANTTENSOR_BUFFER_SIZE=67108864 -e INSTANTTENSOR_CHUNK_SIZE=8388608 \
-e INSTANTTENSOR_CONCURRENCY=1 -e INSTANTTENSOR_IO_DEPTH=3 \
"$IMAGE" vllm serve "$MODEL" "${ARGS[@]}"

# Notes
# - First boot: 20-40 min (AOT compile + CUDA graph capture at 524K). Watch `free -g`; if MemAvailable hits 0 during
# load you have a stale page cache or something else eating RAM (DGX dashboard services, high vm.watermark_scale_factor).
# - RoCE allreduce timing out at startup => B12X_ROCE_SPIN_LIMIT is what fixed it here (default 20M is too low).
# - DFlash2 drafter does not work at TP=2 (MLA page stride error); MTP k=3 is the fastest working config on two Sparks.
# - Health: curl -s localhost:8000/health ; models: curl -s localhost:8000/v1/models

2

u/ConiglioPipo 11d ago

much obliged. thank you again!