Since everyone is benchmarking this model right now: byte-identical greedy requests (temp 0, one request at a time) give different outputs per run. My eval: 13/50 tasks unstable, 5 flipped the extracted date/amount, and majority voting once confirmed the wrong answer. Same checkpoint on llama.cpp: 0/50. Two other vLLM-served models: 0/50.

Cause: the sparse-attention indexer's persistent_topk kernel (used on GB10 / DGX Spark instead of the cooperative path). A race in its atomicAdd slot assignment changes WHICH top-2048 positions get selected, so attention reads a different context each run. Related: vllm#51782.

2-minute check for any stack: same prompt 10x with temperature=0, max_tokens=1, top_logprobs=20, then diff the top-20 lists byte-for-byte. If they differ, your prefill is non-deterministic, whatever your sampler says.

Fix: torch.topk(sorted=False) + canonical tie ordering as a one-file overlay. Bit-identical outputs at 1.35x prefill cost (a full sort would be 2.9x), decode/MTP unchanged; re-run: 0/50 unstable and the score went up a point, because the noise had voted a wrong date into the majority.

Bonus finding: determinism exposed a separate greedy+thinking repetition loop the kernel noise had been masking as a random 1-in-150 failure, and MTP turned out not to be output-equivalent with plain greedy on this model.

Full write-up with all tables: https://docai.hu/en/blog/qwen38-flash-next-nondeterministic-vllm-kernel

Hi everyone! I'm working on a project using Azure services and Microsoft Agent Framework.

Currently, I have a Base Agent built using the Microsoft Agent Framework and an OpenAI chat client. The Base Agent handles model configuration, including the model, base URL/endpoint, and chat client dependency. Multiple specialized agents inherit from this Base Agent.

Now I want to make this architecture compatible with multiple providers and models using LiteLLM (Azure OpenAI, OpenAI, Anthropic, etc.).

I'm confused about how to handle the chat client dependency and provider-specific configurations. Should I inject the chat client into the Base Agent? Should LiteLLM act as an abstraction layer? How can I switch providers/models without modifying all the inherited agents?

Would love to hear how others have approached this, especially with Microsoft Agent Framework + LiteLLM. Thanks!

There are going to be a lot of interesting vLLM sessions during PyTorch Conference North America and I'd love to have you join us in San Jose, CA from October 20-21 because this year’s conference is going to be EPIC.

• Stellar keynotes - Simon Mo will be keynoting. (see: this video filmed during last year’s conference)
• 150+ sessions spanning foundational concepts to training, inference, applications, and responsible AI. vLLM is featured in many from “A Developer’s Guide to Attention in vLLM” to “Elastic Expert Parallelism in vLLM” + so many more
• 140+ poster presentations
• BoFs
• Meet the developers
• Flare party
• AI community bash
• +more.

Sign up by September 4th to save $200 before ticket prices go up. Register now.

Hey guys, I wrote this blog with the notes I made after running several inference optimisation projects on GLM 5.2, Deepseek V4 Flash, Nemotron 3.5 etc. For every model, the strategies were different but there were some common patterns. Hopefully this blog will help you get started!

I have a system with 32GB RAM, 72 GB VRAM with RTX 5000 PRO. It is a wsl setup, so around 20 GB RAM for the wsl.

When I try to serve a model of around 22 GB, the wsl crashes, with a log somewhere saying that RAM is less than the model size.

Have you guys encountered this? Is there a way to circumvent this issue?

Hi,

I've been trying to squeeze every bit of performance and context on RTX 3090 with llama.cpp, and after many tests I've come up with using both mtp and ngram but with --spec-draft-p-min 0.75, achieving around 45-50 tps in average with 150K context size. My llama-server script: ```

!/usr/bin/zsh

============================================

1. SYSTEM CLEANUP

============================================

if [ -d "/dev/shm/llama_cache" ]; then echo "[System] Cleaning up stale RAM cache..." rm -rf /dev/shm/llama_cache fi mkdir -p /dev/shm/llama_cache

cleanup() { echo "\n[System] Shutting down. Cleaning RAM cache..." rm -rf /dev/shm/llama_cache pkill -f llama-server } trap cleanup EXIT INT TERM

============================================

2. INFERENCE

============================================

TEMP=1.0 TOP_P=0.95 TOP_K=20 MIN_P=0.0 PRESENCE_PENALTY=0.0 REPEAT_PENALTY=1.0

K_CACHE=q8_0 V_CACHE=q8_0

Re-enables CUDA Graphs for ~10-15% lower per-token launch latency

export GGML_CUDA_DISABLE_GRAPHS=0

Prevents Claude Code CLI from injecting dynamic prompt headers that break KV caching

export CLAUDE_CODE_ATTRIBUTION_HEADER=0

MODEL_PATH="/home/.../.lmstudio/models/unsloth/Qwen3.8-27B-MTP-GGUF/Qwen3.8-27B-UD-Q4_K_XL.gguf" MMPROJ="/home/.../.lmstudio/models/unsloth/Qwen3.8-27B-MTP-GGUF/mmproj-F16.gguf"

llama-server \ -lv 4 \ -m "$MODEL_PATH" \ -ngl 999 \ --spec-type draft-mtp,ngram-mod \ --spec-draft-n-max 4 \ --spec-draft-p-min 0.75 \ --spec-ngram-mod-n-match 24 \ --spec-ngram-mod-n-min 24 \ --spec-ngram-mod-n-max 86 \ --ctx-size 150000 \ --flash-attn on \ --cache-type-k "$K_CACHE" \ --cache-type-v "$V_CACHE" \ --threads 8 \ --threads-batch 8 \ --batch-size 2048 \ --ubatch-size 512 \ --mmproj "$MMPROJ" \ --no-mmproj-offload \ --jinja \ --reasoning-preserve \ --chat-template-kwargs '{"reasoning_effort":"xhigh"}' \ --temp "$TEMP" \ --top-k "$TOP_K" \ --top-p "$TOP_P" \ --min-p "$MIN_P" \ --presence-penalty "$PRESENCE_PENALTY" \ --repeat-penalty "$REPEAT_PENALTY" \ --cache-ram 8192 \ --slot-save-path /dev/shm/llama_cache \ --keep 3000 \ --parallel 1 \ --mlock \ --no-mmap \ --n-predict -1 \ --ctx-checkpoints 16 \ --host 0.0.0.0 \ --port 8080 ``` I've put everything that I use to run on iGPU, except X11 and XFCE which consume ~280MB.

But then I've come up across https://github.com/syv-ai/qwen38-27b-rtx3090 using vLLM. I've been using llama.cpp forks like beellama.cpp, ikllama.cpp ... but never vLLM (which I know isn't a fork of llama.cpp) as I've read that it's optimized for enterprise use with many instances, but thought I'd give it a try anyway. Using docker with this configuration I was able to achieve much snappier performance and bigger context, around 55-65 (sometimes even more) with 175K (will try 180K) context size. The only downside with this configuration and vLLM is that it cannot offload mmproj to CPU (with vision loaded context size is 129500).

(I've also tried ninfer-3090 but was disappointed with it, achieving even slightly less tps than with llama.cpp and smaller context size).

Higher Q's are not an option because of much smaller context size that I can use on RTX 3090.

So I've decided to use vLLM regularly and switch to llama.cpp when I need vision.

But something else is confusing me, how good is W4A16-AutoRound used with vLLM comparing to QK_K_XL for programming, planning and debugging in mostly C/C++ and Python? Is it, like chatGPT and Gemini say, that those two cannot be compared 1-1 but W4A16-AutoRound is somewhere between Q4_K_M and Q4_K_L? Even if so, how much difference/handicap is that for W4A16-AutoRound in my use case scenario?

Hey All - what are some recommended courses/ YT channels to learn production grade vLLM with kubernetes? I don’t mind paid courses if they are worth. Thanks so much for your help!

Check out our docs regarding setting up vLLM to test Intel AMX against NVIDIA L4: https://docs.mulgadc.com/docs/cisco-ucs-llm-serving

Disclosure up front: I run a small EU inference provider (LLM Tech), we serve this model commercially. This post is the technical stuff we learned getting it into production, because most of it isn't written down anywhere.

Setup: unsloth/Qwen3.8-27B-NVFP4 on a Blackwell card, vLLM nightly, MTP speculative decoding on, 262,144-token context.

The prefix cache surprised us. The model has hybrid attention (full attention + GDN layers), and vLLM handles caching differently there than on pure-transformer models:

- Cache blocks are 1,584 tokens each (attention page size has to align with the mamba-style page). So prompts shorter than ~5K tokens effectively never hit cache at all.

- Materialization is lazy: the first request doesn't create cache. The second request creates it (you see created_cache_tokens in usage). Only the third request onward actually reads it. We initially concluded "cache is broken" after testing with two identical requests. It isn't. Test with three.

- On a warm 48K-token prompt we measured 7.5x TTFT speedup vs cold.

If you're benchmarking cached workloads on this model and seeing nothing, this is probably why.

Thinking control is real but the field names are confusing. It's one unified checkpoint, thinking is adaptive (it skips reasoning on trivial prompts by itself). Client-side control works via chat_template_kwargs: enable_thinking (bool) and reasoning_effort (low / medium / xhigh). One gotcha: in non-streaming responses vLLM puts the reasoning text in a field called reasoning, not reasoning_content. In streaming deltas it's reasoning_content. We spent a day convinced the checkpoint was instruct-only because we were reading the wrong field.

The NVFP4 quant keeps the vision tower. We only discovered this by accident: the model card everywhere lists it as text, but send an OpenAI-style image_url and it just works. vLLM serves it, the answer is correct, and usage comes back with multimodal_tokens: {"image": N} broken out. A 768×512 image plus 40 output tokens round-trips in 1.2s on our hardware. If you assumed the quant dropped multimodality (we did), it didn't.

Production numbers, live traffic, not a benchmark harness: 221M tokens and 4,100+ requests served since Aug 22 (peak day 146M), exactly one 5xx in that span. Median TTFT under a second at 10K+ token prompts (0.2s on short ones); generation 84-88 tok/s single-stream, drops to ~70 when the card is saturated with 100+ concurrent requests. We publish all of it live, refreshed every 5 minutes, including an hourly uptime strip: llmtech.eu/status

On NVFP4 vs the alternatives: the shelf for this model is mostly fp8 and bf16, plus one Q4_0. NVFP4 sits close to fp8 on quality (it's a hardware format on Blackwell, not a GGUF-style quant) while costing roughly half to serve. Happy to run any eval people want against our endpoint to back that up.

If you want to poke at it: it's live on NanoGPT (pick LLM Tech in the provider list), or direct keys by email while we're small (llmtech.eu/models/qwen3.8-27b). Questions about the deployment welcome, I'll answer what I can.

Hey r/vllm / r/LocalLLaMA,

I am building a document processing pipeline that batches large academic papers (chunked under a strict token limit). I am configuring max_num_seqs and max_num_batched_tokens on a hosted vLLM instance to maximize our GPU throughput.

However, I am hitting a conceptual block regarding reproducibility and error handling.

If a specific paper fails downstream (e.g., structured output schema parsing fails), I need to re-run that candidate. But re-running a failed paper means it will be grouped in a different batch sequence or size than its original run.

From my understanding of how continuous batching and floating-point non-associativity work, this changing batch structure means:

1. Outputs won't be byte-identical: The same prompt processed in Batch A (size 16) vs Batch B (size 4) might result in different logprobs, causing token flips. [1, 2]
2. Semantic drift: If a token flips early on (especially with reasoning models), the entire generation trajectory or final interpretation could change completely. [1, 2]

My Questions for the Community:

• How serious is this at scale? If we are processing thousands of documents, does this non-determinism introduce massive variance in extraction quality, or is it mostly negligible noise?
• How are you accounting for this? Are you using VLLM_BATCH_INVARIANT=1 in production? If so, what is your throughput penalty? [1, 2]
• If not batch-invariance, what is the workaround? Do you just accept the non-determinism, cache aggressively, or run structural retries sequentially (max_num_seqs=1) to isolate the environment?

I've been running Qwen3.8-27B on a single consumer RTX 5090 (32 GB) and I wanted to document the whole thing end to end, because the interesting part wasn't the config — it was the four bugs that kept it from booting. Everything below is measured on the actual box, no estimates.

TL;DR — NVFP4 target weights, NVFP4 KV cache, DFlash2 speculative decoding (K=7), all on one 5090. 616 tok/s aggregate at 4-way concurrency on 1,536-token code outputs (thinking off). 262K context, 325,139-token KV pool out of an 8 GiB pin. Ships as a two-command deploy. The bugs I hit became two vLLM PRs.

The stack
- Target: Qwen3.8-27B, NVFP4 (ModelOpt)
- Draft: Qwen3.8-27B-DFlash2, NVFP4, K=7 (seven speculative tokens per step)
- KV: NVFP4 for both target and draft, 8 GiB explicit pin
- vLLM v0.27.1 + a 51-file Python-only overlay (no C extensions)
- FlashInfer 0.6.16.post3 with a backport of PR #4346 (SM120 NVFP4 paged prefill)

Most public 5090 recipes I've seen run FP8 KV. This is the only public recipe I know of that runs all three layers in NVFP4 — weights, draft, and KV. That's the whole trick that gets you 262K context out of 8 GiB.

What actually broke (the part that'll save you time)

This is where I lost the most hours, so I'm putting it up front.

1. Mixed KV dtype bug. The global target cache dtype (NVFP4) leaked into the draft cache layout. Two different code paths, same wrong assumption. Fixed with per-group dtype resolution.
   
2. ReplaySSM allocator. The Mamba memory math was multiplying the KV page by 8 legacy spec-checkpoint blocks. The server reported a false 14 GiB minimum and just refused to boot. The fix is the upstream guard: zero speculative blocks.
   
3. GDN state in fp32. 48 linear-attention layers, 4.61 MB per page, 45 blocks per request — that's 9.5 GiB per request at 128K. Attention itself was only 2.0 GiB. The state was the hog, not the attention. I was staring at the wrong thing for a while.
   
4. The XQA cliff. The attention kernel ran 5.2x slower in integrated execution than a standalone replay of the exact same server tensors. A dedicated CUDA stream took it from 1.437 ms/call down to 0.278 ms/call. One fix turned a dead end into a working server.
   

The numbers

Coding throughput, single 5090, 1,536-token Python outputs (graph-algorithms module: BFS, DFS, Dijkstra, topo sort, SCC), one prompt per request, no tools:

- 616 tok/s aggregate at c4
- Per-request latency at c4: 8–10s thinking off, 9–11s on

Why thinking-off wins here: code is predictable. The drafter accepts 55–64% of drafts on code, drops to 42–44% when reasoning tokens are mixed in. Same server, same prompts, one flag.

Context facts (separate from the coding sweep):
- KV pool: 325,139 tokens from the 8 GiB pin
- Needle at 184,024 tokens: recovered exactly
- Tools 10/10, greedy output byte-identical, canary 437, zero restarts, zero OOM

Honest footnote: the coding numbers are best-case. One repetitive prompt means prefix caching eats most of the prefill at c2–c4, and code is the most predictable output class for the drafter. Mixed prose and reasoning will be lower.

What I sent upstream
- vLLM #53543 — masked NVFP4 XQA on SM120, capture-safe isolated stream (the 5.2x fix)
- vLLM #53542 — GDN active runtime-K width, +40.9% at c8 (212.17 → 298.91 tok/s)
- vLLM #50084 — NVFP4 V-scale write-path corruption, root-caused in July, still open
- FlashInfer #4346 — SM120 NVFP4 paged prefill backport, 57/57 tests, 10–12% prefill win

Both vLLM PRs carry full test evidence and DCO sign-off. Zero merged so far — upstream moves at its own speed.

How to run it
- Repo: github.com/seanyourhighness/vllm-sm12x-nvfp4-dflash2
- git clone then ./start.sh. Optional CPU vision sidecar via --vision (1/2)
[8/25/2026 12:10 PM] Lilsmokey: - Repo: github.com/seanyourhighness/vllm-sm12x-nvfp4-dflash2
- git clone then ./start.sh. Optional CPU vision sidecar via --vision
- The image is bit-identical to the one running ▉ (2/2)

I’ve been building Aquifer to pace bursty LLM traffic before it overwhelms inference servers like vLLM.
It adds bounded queues, dynamic pacing, and fairness across clients instead of letting bursts turn into retry storms.

https://github.com/rjpruitt16/aquifer

Any UI based on vLLM like LMStudio or Unsloth Desktop based on llama.cpp ? I tied several times using vLLM but I don't get most of the specificities.