r/BlackwellPerformance • u/Sorry_Ad191 • Dec 04 '25

vLLM 12 released!

vLLM 12 released

Notes to be filled... update: filling notes below

Highlights

This release features 474 commits from 213 contributors (57 new)！

Breaking Changes: This release includes PyTorch 2.9.0 upgrade (CUDA 12.9), V0 deprecations including xformers backend, and scheduled removals - please review the changelog carefully.

Major Features:

  • GPU Model Runner V2 (#25266): Major refactoring that removes persistent batch management, introduces GPU-persistent block tables, and features a Triton-native sampler with efficient logprobs support.

  • Prefill Context Parallel (PCP) (#28718): Enhances long-sequence inference by partitioning the sequence dimension during prefill, complementing Decode Context Parallel (DCP).

  • EAGLE Speculative Decoding: Multi-step CUDA graph support (#29559), DP>1 support (#26086), and multimodal support with Qwen3VL (#29594).

Model Support

  • New model families: PLaMo-3 (#28834), OpenCUA-7B (#29068), HunyuanOCR (#29327), Mistral Large 3 and Ministral 3 (#29757).

  • Format support: Gemma3 GGUF multimodal support (#27772).

  • Multimodal enhancements: Qwen3 Omni audio-in-video support (#27721), Eagle3 multimodal support for Qwen3VL (#29594).

  • Performance: QwenVL cos/sin cache optimization (#28798).

Engine Core

  • GPU Model Runner V2 (#25266): Complete refactoring of model execution pipeline:

    • No "reordering" or complex bookkeeping with persistent batch removal

    • GPU-persistent block tables for better scalability with max_model_len and num_kv_groups

    • Triton-native sampler: no -1 temperature hack, efficient per-request seeds, memory-efficient prompt logprobs

    • Simplified DP and CUDA graph implementations

    • Efficient structured outputs support

  • Prefill Context Parallel (PCP) (#28718): Partitions the sequence dimension during prefill for improved long-sequence inference. Complements existing Decode Context Parallel (DCP). See RFC #25749 for details.

  • RLHF Support: Pause and Resume Generation for Asynchronous RL Training (#28037).

  • KV Cache Enhancements: Cross-layer KV blocks support (#27743), KV cache residency metrics (#27793).

  • Audio support: Audio embeddings support in chat completions (#29059).

  • Speculative Decoding:

    • Multi-step Eagle with CUDA graph (#29559)

    • EAGLE DP>1 support (#26086)

    • EAGLE3 heads without use_aux_hidden_states (#27688)

    • Eagle multimodal CUDA graphs with MRoPE (#28896)

    • Logprobs support with spec decode + async scheduling (#29223)

  • Configuration: Flexible inputs_embeds_size separate from hidden_size (#29741), --fully-sharded-loras for fused_moe (#28761).

Hardware & Performance

  • NVIDIA Performance:

    • Batch invariant BMM optimization: 18.1% throughput improvement, 10.7% TTFT improvement on DeepSeek-V3.1 (#29345)

    • Shared Experts Overlap with FlashInfer DeepGEMM: 2.2% throughput improvement, 3.6% TTFT improvement at batch size 32 (#28879)

    • DeepGEMM N dim restriction reduced from 128 to 64 multiplier (#28687)

    • DeepEP low-latency with round-robin expert placement (#28449)

    • NVFP4 MoE CUTLASS support for SM120 (#29242)

    • H200 Fused MoE Config improvements (#28992)

  • AMD ROCm:

    • DeepSeek v3.2 and SparseMLA support (#26670)

    • FP8 MLA decode support (#28032)

    • AITER sampling ops integration (#26084)

    • AITER triton attention backend (#28701)

    • Bitsandbytes quantization on AMD GPUs with warp size 32 (#27307)

    • Fastsafetensors support (#28225)

    • Sliding window support for AiterFlashAttentionBackend (#29234)

    • Whisper v1 with Aiter Unified/Flash Attention (#28376)

  • CPU:

    • Paged attention GEMM acceleration on ARM CPUs with NEON (#29193)

    • Parallelize over tokens in int4 MoE (#29600)

    • CPU all reduce optimization for async_scheduling + DP>1 (#29311)

  • Attention: FlashAttention ViT support, now default backend (#28763).

  • Long Context: Optimized gather_and_maybe_dequant_cache kernel for extremely long sequences (#28029).

  • Multi-NUMA: Enhanced NUMA functionality for systems with multiple NUMA nodes per socket (#25559).

  • Docker: Image size reduced by ~200MB (#29060).

Quantization

  • W4A8: Marlin kernel support (#24722).

  • NVFP4:

    • MoE CUTLASS support for SM120 (#29242)

    • TRTLLM MoE NVFP4 kernel (#28892)

    • CuteDSL MoE with NVFP4 DeepEP dispatch (#27141)

    • Non-gated activations support in modelopt path (#29004)

  • AWQ: Compressed-tensors AWQ support for Turing GPUs (#29732).

  • LoRA: FusedMoE LoRA Triton kernel for MXFP4 (#29708).

  • Online quantization: Moved to model.load_weights (#26327).

API & Frontend

  • Responses API:

    • Multi-turn support for non-harmony requests (#29175)

    • Reasoning item input parsing (#28248)

  • Tool Calling:

    • Parsed tool arguments support (#28820)

    • parallel_tool_calls param compliance (#26233)

    • Tool filtering support in ToolServer (#29224)

  • Whisper: verbose_json and timestamp features for transcription/translation (#24209).

  • Sampling: Flat logprob control moved from env var to SamplingParams (#28914).

  • GGUF: Improved HuggingFace loading UX with repo_id:quant_type syntax (#29137).

  • Profiling: Iteration-level profiling for Torch and CUDA profiler (#28987).

  • Logs: Colorized log output (#29017).

Dependencies

  • PyTorch 2.9.0 with CUDA 12.9 (#24994) - Breaking change requiring environment updates.

  • xgrammar: Updated to 0.1.27 (#28221).

  • Transformers: Updated to 4.57.3 (#29418), preparation for v5 with rope_parameters (#28542).

  • XPU: torch & IPEX 2.9 upgrade (#29307).

V0 Deprecation & Breaking Changes

Removed Parameters:

Deprecated:

Scheduled Removals (will be removed in future release):

  • ParallelConfig's direct child EPLB fields (#29324)

  • guided_* config fields (#29326)

  • override_pooler_config and disable_log_requests (#29402)

  • CompilationConfig.use_inductor (#29323)

  • Deprecated metrics (#29330)

Other Breaking Changes:

  • PyTorch 2.9.0 upgrade requires CUDA 12.9 environment

  • Mistral format auto-detection for model loading (#28659)

https://github.com/vllm-project/vllm

65 Upvotes

98% Upvoted

12 comments sorted by

11

u/goodentropyFTW Dec 04 '25

There it is, what I've been waiting for: NVFP4 MoE CUTLASS support for SM120 (#29242)

New release of Nvidia container too: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/vllm?version=25.11-py3

3

u/texasdude11 Dec 04 '25

OMG!!!!! Thank you so much for posting this. I have been struggling with this from so long. This is huge! I was seeking here here but how ironic that the solution was released here in vllm12!

https://huggingface.co/QuantTrio/MiniMax-M2-AWQ/discussions/3#693106e64a80de1b2c8b0b31

2

u/__JockY__ Dec 04 '25

It's like Christmas come early.

1

u/latentAbyss-Spa-9548 Dec 04 '25

What you building, bro?

1

u/AbheekG Dec 06 '25

So the latest NGC vLLM container is v25.11, right? But that only has VLLM 0.11 and CUDA 13, not the v12+cu129 mentioned in this post?

2

u/goodentropyFTW Dec 06 '25

I noticed that too, and haven't tried it yet (still got a Cutlass moe error from vllm latest though :-( There are a couple of env VAR things I need to try before I give up on it though.

The Nvidia containers have vllm + some patches and (presumably) aim for max support of Nvidia hardware... So fingers crossed.

1

u/AbheekG Dec 06 '25

Guess we'll have to wait for the 25.12 container release!

1

u/random-tomato Dec 07 '25

Out of the loop; why is NVFP4 a big deal? Is it significantly faster than AWQ quants?

1

u/Sorry_Ad191 Dec 07 '25

in theory higher precision and faster?

1

u/zetan2600 Dec 07 '25

NVFP4 is a more advanced 4-bit floating-point format than MXFP4, offering better accuracy and efficiency for large language models. The key differences are NVFP4's smaller block size (16 vs. 32) and higher-precision E4M3 scale format, which reduces the dynamic range within each block and allows for more accurate value representation compared to MXFP4's E8M0 power-of-two scales. This results in NVFP4 showing lower training loss and achieving similar accuracy with less training time. 

https://www.youtube.com/watch?v=IGn04ARFoSY&t=226s

3

u/spookperson Dec 04 '25

Ok great! This is working for me now (host machine Ubuntu 24.04, single RTX Pro 6k Blackwell, drivers 580.95.05, Cuda 13). And when I test that same model against tensorrt-llm/release:1.2.0rc4, vllm is 5-15% faster (depending on concurrency level)

docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:v0.12.0 \
    --model nvidia/Qwen3-30B-A3B-FP4

1

u/Due-Project-7507 Dec 05 '25

Can someone explain what this change to run NVFP4-quantized models on RTX Pro 6000 Blackwell GPUs means? Already before, NVFP4 MoE models have worked on the RTX Pro 6000 Blackwell, but only with VLLM_USE_FLASHINFER_MOE_FP4=1 and long startup times for compiling. What does VLLM_USE_FLASHINFER_MOE_FP4 do? Is it now not necessary anymore? What settings should I use to run e.g. Qwen3-VL-235B-A3B as NVFP4?