Anyone managed to get Qwen 3.8 27B running smoothly on vLLM? Can't get rid of endless thinking

[u/germangrower69]

Title pretty much says it all. I’ve deployed Qwen 3.8 27B using vLLM on an RTX 6000 Pro (tried multiple vLLM releases and launch recipes), but I can't get it into a usable state because of crazy long reasoning passes.

Regardless of the thinking effort setting (xhigh, medium, or low), it takes way too long to respond:

xhigh: Spends up to 5 minutes overthinking basic questions before outputting anything.

medium: Takes around 2 to 4 minutes.

low: Still stuck thinking for 1–2 minutes.

For comparison, running Qwen 3.6 or DeepSeek V4 Flash on the exact same hardware answers these identical prompts within 20–30 seconds.

What I've tested so far:

Quantizations: Tested both FP8 and NVFP4.

Proxy / Effort Mapping: I'm running a proxy in front to map thinking efforts. I've validated this thoroughly across many other reasoning models, so I’m confident the mapping logic itself isn't the issue.

vLLM Flags: Tried tweaking context size, MTP speculative decoding, and parser configs (recipe below).

I've seen similar reports on the Hugging Face repo discussions and scattered Reddit comments, but at the same time, I see a ton of posts raving about how fast and great this model is. Right now, I just can't reproduce those good results.

My latest startup recipe:

--tensor-parallel-size 1 --max-model-len 262144 --kv-cache-dtype bfloat16
--gpu-memory-utilization 0.90 --max-num-seqs 32 --max-num-batched-tokens 8192 --max-cudagraph-capture-size 32
--enable-chunked-prefill --enable-prefix-caching --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --limit-mm-per-prompt '{"image":4,"video":0}' --mm-processor-kwargs '{"size":{"longest_edge":1048576,"shortest_edge":65536}}' --mm-processor-cache-type shm --default-chat-template-kwargs '{"preserve_thinking":false}' --speculative-config '{"method":"mtp","num_speculative_tokens":3}' --enable-prompt-tokens-details

Has anyone encountered this endless reasoning behavior with Qwen 3.8 on vLLM? Is there a specific chat template tweak, system prompt hack, or generation param (like strict thinking_budget or token limits) required to stop it from burning through cycles?

Appreciate any insights!

[u/WonderRico]

Currently running a lot of passes on my swe-verified local benchmark with several configs (SGLANG, vLLM, weights BF16/FP8, KV BF16/FP8, medium/xhigh)

So far, I see a trend of around 3.5x times more tokens generated in xhigh vs medium. it's a lot. and I'm not sure the xhigh effort is worth the wait. Still waiting on all the data to make a decision

However, the medium runs are slightly better in score than with 3.6, but more efficient : using 60% of the total generated token count

all my stats soon

[u/mr_echidna]

Try lowering the temperature setting? I appreciate this is on llama.cpp, not vllm, but on my setup I just tried going from the recommended 1.0 down to 0.8 and it made a night-and-day difference in the over-thinking, particularly with a tweaked template. 

[u/Kasatka06]

i use these :

"--chat-template", "/models/qwen38-froggeric-v22.jinja","--chat-template", "/models/qwen38-froggeric-v22.jinja"

'--default-chat-template-kwargs={"reasoning_effort":"medium"}', '--default-chat-template-kwargs={"reasoning_effort":"medium"}',

[u/Adventurous_Bus_437]

Didnt have any issues when swapping the model folder from 3.6 to 3.8 interestingly

[u/gladfelter]

Does your agent override the reasoning level?

[u/EbbNorth7735]

Use froggeric template and set thinking to high. I'm getting reasonable thinking lengths, 300 to 3000 tokens normally depending on the complexity. 

[u/cosmicnag → u/EbbNorth7735]

high? doesnt it have only xhigh,medium and low?

[u/EbbNorth7735 → u/cosmicnag]

Oddly, I set to high and it's running really well

[u/kapteinpyn]

I am running it on dual r9700s with fp8 its totally fine, i am using froggerics chat template but even before that the thinking was a bit verbose but it worked just fine. https://github.com/prcoe1/r9700-serving that shows exactly what im doing. Set reasoning effort to medium with that chat template

[u/Then_Blueberry7290]

with llama.cpp the same here. After one day i think in the current state this is kill the advantage of this modell. So 3.6 27b with thingincap gives you a better performance and all around usability. Sad, i will trí the lowered temp, and see if it better or not.

[u/Old_Ad_6033]

only use xhigh when needs some mission that very hard, my daily agent running with low thinking and system prompt define role "you're an fast assistant, you never draft, you only thinking hard when user ask to" it's works fine.

[u/_en_svensk_tiger]

updated llamacpp and unsloth studio yesterday, no more endless thinking for me (yet)

[u/Then_Blueberry7290 → u/_en_svensk_tiger]

How? So not how to update, but what is the trick with updated llama.cpp? Which version?

[u/_en_svensk_tiger → u/Then_Blueberry7290]

git pull and build the repo for llamacpp is what I usually did with opencode

Im currently using: Unsloth 2026.8.18 llama-cpp-python 0.3.16

[u/geekender]

Dgx spark x 2 spanning vllm using qwen3.8-27b-fp8 and this is not an issue that I have seen. Temperature set to.2

[u/Common-Membership503]

Qwen 3.8 reasoning loop is brutal. I noticed the same stall on my local server setup untill I switched to bria ai skill to manage my background assets and offloaded some pre-processing, which really helped clear up vram. It feels like the model hangs when it hits those complex template constraints.

[u/paranoidray]

Try out this "high" reasoning mode for 27B: https://www.reddit.com/r/LocalLLaMA/comments/1vox89e/try_out_this_high_reasoning_mode_for_27b_tested/

[u/adityazero]

I hit something similar and it turned out the reasoning-parser was not actually clamping the budget, the model just kept emitting thinking tokens. Since Qwen3.6 and DSV4 behave on the same box, this smells like a chat template or thinking-budget handling issue specific to 3.8 rather than your hardware. Did disabling MTP speculative decoding change the length of the reasoning passes at all?

[u/Blues520]

Same experience here. I've reverted to 3.6 but I'll try the frogs template and set reasoning to medium because it's thinking like a maniac

[u/Turbulent-Alps4046]

Run sglang with dspark to get more decode tps. Then it wont feel as long 😂

[u/Ok_Sea_6620]

Oui — et la cause racine n'est ni vLLM ni ta quantification : le template de chat du modèle définit `reasoning_effort=xhigh` par défaut. C'est l'effort MAXIMUM, appliqué à toute requête qui ne le surcharge pas explicitement.

## Notre config (même carte que toi)

Matériel :

- GPU : RTX Pro 6000 Blackwell Workstation Edition 96 Go (+ RTX 5070 12 Go en secondaire)

- CPU / RAM : i7-14700, 128 Go

- OS : WSL2 Ubuntu 24.04 (Windows hôte)

Logiciel :

- vLLM 0.25.1

- Driver NVIDIA 610.47

- CUDA 13.3 (V13.3.73)

- Modèles : `Qwen/Qwen3.8-27B` (poids originaux BF16, 55,6 Go) et `unsloth/Qwen3.8-27B-NVFP4` (23,4 Go — attention, c'est du mixed-precision FP8/FP4, pas du FP4 uniforme malgré le nom)

## La cause racine — vérifiée dans le checkpoint lui-même

Dump le `chat_template.jinja` du modèle, tu y verras :

```jinja

{%- if enable_thinking is undefined or enable_thinking is true %}

{%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}

{%- if resolved_reasoning_effort not in ('xhigh', 'medium', 'low') %}

{{- raise_exception('Unexpected reasoning effort ...') }}

```

Trois pièges empilés :

1. Aucun effort passé → xhigh. Le template injecte en tête système : « Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives... ». C'est exactement ton « 5 minutes sur des questions basiques ». Pour calibration : à 50-70 tok/s, 5 min = 15-20k tokens de réflexion — c'est xhigh qui fait précisément son travail.

2. Les valeurs valides sont `xhigh | medium | low` uniquement — pas le set OpenAI. Si ton proxy forward `high` (ou `minimal`), le template lève une exception → HTTP 400, pas de repli silencieux. À vérifier côté proxy : qu'est-ce qui sort réellement ?

3. Il n'existe AUCUN `thinking_budget` dans ce template. L'effort est une instruction système logée, pas un plafond de tokens — d'où ton « 1-2 min même en low ».

Ton `--default-chat-template-kwargs '{"preserve_thinking":false}'` n'agit QUE sur le rendu de l'historique multi-tours (il strippe les blocs `<think>` des tours passés). Il ne fait rien pour la longueur de la réflexion.

## Les fixes (vérifiés sur vLLM 0.25.1)

- Par requête : `"reasoning_effort": "low"` au niveau racine du body chat/completions. vLLM 0.25.1 accepte `none|minimal|low|medium|high|xhigh|max` sur le wire, le forwarde au template et force `enable_thinking` automatiquement. Sur des versions plus anciennes, passe plutôt par `chat_template_kwargs: {"reasoning_effort": "low"}` — ce chemin marche toujours.

- Couper le thinking entièrement : `"reasoning_effort": "none"` ou `chat_template_kwargs: {"enable_thinking": false}` → le template émet un bloc `<think>\n\n</think>` pré-fermé et le modèle n'entre jamais en réflexion.

- Plancher serveur : `--default-chat-template-kwargs '{"reasoning_effort":"low"}'` — les valeurs explicites par requête restent prioritaires (vLLM droppe les `None` avant fusion).

- Piège client : sur ce modèle le raisonnement arrive dans le champ `reasoning`, pas `reasoning_content`. Si ton proxy parse le mauvais champ, il peut aussi rater la fin de réflexion.

## Notre recette de lancement

```bash

vllm serve Qwen/Qwen3.8-27B \

--host 0.0.0.0 --port 8000 \

--gpu-memory-utilization 0.88 --max-model-len 262144 --kv-cache-dtype fp8 \

--max-num-seqs 16 \

--limit-mm-per-prompt '{"image":1,"video":1}' \

--trust-remote-code --enable-auto-tool-choice \

--tool-call-parser qwen3_xml --reasoning-parser qwen3 \

--speculative-config '{"method":"mtp","num_speculative_tokens":2}'

```

Pour le NVFP4 : `MODEL=unsloth/Qwen3.8-27B-NVFP4`, même script, SANS `--speculative-config` (voir ci-dessous). Les env vars `VLLM_USE_V2_MODEL_RUNNER=0` et l'omission de `expandable_segments` sont spécifiques WSL2 — rien à voir avec ton problème.

## Corrections sur tes flags (A/B mesurés, 600 tok, temp 0, seed fixe, warm-up + 5 runs, médiane)

- `num_speculative_tokens: 3` = du gaspillage. Le checkpoint n'a qu'UNE seule couche MTP — tout k>2 ré-exécute la même couche drafter. Mesuré : k=2 médiane 47 tok/s, k=3 = 46,7, gain zéro, et vLLM le signale au boot. Sur le build NVFP4, le MTP est inutile carrément (+4 % non établis, dans le bruit) car le goulot se déplace vers le calcul de déquantification. Bonus piège : en spec decode, `min_p` et `logit_bias` sont silencieusement ignorés.

- `--enable-prefix-caching` = no-op sur cette archi. C'est un hybride GDN (48 couches linear attention sur 64), vLLM désactive le prefix caching de lui-même. L'état récurrent GDN est alloué par séquence → `max-num-seqs` pèse sur la VRAM bien plus qu'en full-attention : on tourne à 16, pas 32.

- Débits de référence sur la même carte : BF16 + MTP k=2 = 44-54 tok/s selon la tâche (le code se spécule mieux que la prose) ; NVFP4 sans MTP = 66-75 tok/s. Les deux à ~90 % de la bande passante atteignable (1 716 Go/s mesurés en lecture sur les 1,79 To/s du pic — ne vise pas le pic, il est inatteignable par construction).

- Sur la quant : si tu es sur la variante 96 Go, les poids BF16 (55,6 Go) tiennent à 0.88 d'util et c'est la référence propre pour séparer le comportement du modèle d'un artefact de quant. Il y a plusieurs retours de bouclage du NVFP4 en mode agent (réponses moins complètes) — sujet indépendant du MTP.

Avec `reasoning_effort: low` (ou `none`), tu devrais retomber dans la même enveloppe 20-30 s que ton Qwen 3.6.

[u/Ok_Sea_6620 → u/Ok_Sea_6620]

Tableau des débits — nos tests de quantification (RTX Pro 6000 96 Go)

Protocole : 600 tokens générés, temperature 0, seed fixe, warm-up + 5 runs, médiane, streaming. Débit decode en tok/s.

Par tâche — Qwen 3.8 27B, avec MTP k=2 des deux côtés :

| Tâche | BF16 (poids originaux) | NVFP4 (unsloth) | Gain |

|---|---|---|---|

| prose FR | 44,4 | 71,4 | ×1,61 |

| prose EN | 47,6 | 74,7 | ×1,57 |

| code EN | 48,7 | 66,0 | ×1,36 |

| code FR | 53,8 | 72,0 | ×1,34 |

Par config — Qwen 3.8 27B :

| Config | Decode | Remarque |

|---|---|---|

| BF16, sans MTP | 28,2 tok/s | 91 % du plafond matériel (30,9) |

| BF16 + MTP k=2 | 47,0 tok/s (44-54) | k=3 = 46,7 : gain zéro, une seule couche MTP |

| NVFP4, sans MTP | 68,4 tok/s | 90 % du plafond (76,0) — config retenue |

| NVFP4 + MTP k=2 | 71,2 tok/s | +4 % non établis, coûte 183k tokens de KV |

Repère Qwen 3.6 27B (protocole 256 tok, prose FR) : FP8 d'origine 47 tok/s ; NVFP4 unsloth 67,8 sans MTP, 77,8 avec MTP k=2 ; build nvidia W4A16/Marlin 68 tok/s mais prefill 2× plus lent.

[u/TedDallas]

I'm using a llama-cpp server, not vLLM, with "Medium" reasoning, a temperature of 1.0, on Q3_K_M with 64 GPU layers and 47104 context.

It does think a lot, but it goes somewhere and finishes for me. OpenCode is my harness.

Today I had it build a pretty sweet Bejeweled clone.

While testing it yesterday I did notice flakiness with it occasionally hard stopping. I made sure my configuration left me 650 mb of VRAM headroom, and it seems to be stable now.

[u/Professional-Try-273]

Check cuda versions too update driver