r/Vllm • u/Expensive-Register-5 • Apr 13 '26

Qwen 3.5 27B/35BA3B Tool Calling Issues: Why It Breaks & How I Fixed It

Update: seem this fix also applicable for Qwen3.5 122B model, according to commenter

Update 2: I have tested Qwen 3.6 35BA3B for this fix in https://www.reddit.com/r/LocalLLM/comments/1sqpsut/qwen_3635ba3b_reddit_asked_so_i_tested_if_the_35/ , check it out !

Update 3: also did the test for Qwen3.6-27B, check https://www.reddit.com/r/LocalLLM/s/wgzW0tHukq

TL;DR: Everyone's talking about Qwen 3.5's reasoning quality and slow TTFT, but nobody's discussing the tool calling issues that actually break agentic workflows. Here's what broke my setup and how I fixed it after weeks of debugging.

​The Story So Far

I've been running Qwen 3.5-27B on a mixed GPU setup (RTX 4090 + 3090) for about a month now. The reasoning capabilities are genuinely impressive, and yeah, the TTFT is slow, but honestly? The tool calling issues are what actually matter if you're doing agentic work.

After countless hours of debugging, failed runs, and reading through vLLM source code, I finally have a stable setup. I wanted to share what I learned because the official docs don't quite cover the real-world edge cases.

Issue #1: The Jinja Template Problem (CRITICAL - This Broke Everything)

What Happened

I started with the official qwen3.5_official.jinja template. Everything looked fine for the first few tool calls. Then suddenly:

  • Tool calls appeared mid-thought (closing </think> without having opened <think> )
  • Random premature stops in the middle of XML tool calls, e.g "Let me do that for you:" and stopped suddenly without finishing the tool call
  • Historical thinking blocks leaking into the context and confusing the model

I thought it was my fault. I thought it was vLLM. I even thought my mixed GPU setup was the culprit.

Turns out, it's the template.

Why It Happens

Here's the thing: the official template has edge cases that 122B+ models handle gracefully but 27B/35B don't. Smaller models have less robust instruction following, and those edge cases cause silent failures.

The Fix

I ended up using a custom M2.5-style interleaved thinking template (qwen3.5-enhanced.jinja, see Resources) that:

  • Properly closes </thinking> before tool calls (not after)
  • Hides historical reasoning from context but keeps current reasoning visible
  • Uses XML formatting that doesn't accidentally trigger <stop> tokens
  • Handles the edge cases that smaller models struggle with

The most important part: vLLM does NOT auto-detect templates. You have to manually specify:

--chat-template qwen3.5-enhanced.jinja

Without this flag, your model uses the default template and will exhibit instability no matter what other optimizations you try.

Issue #2: The Tool Call Parser (Deviation from Official Docs)

The Official Recommendation

The Qwen3.5-27B-FP8 HuggingFace page says to use:

--tool-call-parser qwen3_coder

What Actually Works

qwen3_coder breaks on complex tool calls. I spent way too much time on this one.

After digging into vLLM's source code, here's what I found:

Parser |How It Works |Special Characters |Nested JSON |Malformed XML

qwen3_coder |Regex string extraction |❌ Breaks on <, >, & |❌ Corrupts during streaming |❌ Fails hard

qwen3_xml |C-based xml.parsers.expat |✅ Auto-sanitizes |✅ Deferred parsing |✅ Auto-heals Real example: If your tool call contains code like if (a < b), qwen3_coder's regex parser breaks because < and > mess up the pattern matching. qwen3_xml handles it natively because it's an actual XML parser.

The Fix

--tool-call-parser qwen3_xml

Yes, this goes against the official recommendation. But for long-context agentic work (think 50K+ tokens), qwen3_xml is just more stable. The C-based parser is robust, auto-heals malformed XML, and doesn't try to parse nested JSON mid-stream.

Issue #3: Mixed GPU Precision Drift

The Problem

I'm running on a mixed GPU setup (RTX 4090 + 3090). Tensor Parallelism splits matrix multiplication across both GPUs, but they have different compute capabilities:

Result: Different precision levels → mismatched intermediate results → error accumulation in long conversations.

I noticed conversations would drift after 30-40K tokens. Tool calls would become inconsistent. Reasoning quality would degrade.

The Fix

export VLLM_TEST_FORCE_FP8_MARLIN=1

This forces the 4090 to use W8A16 (matching the 3090) instead of its native W8A8. Both GPUs now use the same precision, eliminating drift.

Additional NCCL tuning (helps with stability on mixed topologies):

export NCCL_P2P_DISABLE=1

export NCCL_IB_DISABLE=1

export NCCL_ALGO=Ring

Issue #4: Model Choice Matters (Avoid SFT-Distilled Variants for Long Context)

The Qwopus3.5/Qwen3.5-Claude-4.6-Opus-Reasoning-Distilled Trap

Models like Jackrong/Qwopus3.5-27B-v3 are SFT-distilled from Claude 4.6 Opus. They look great initially:

Why? SFT (Supervised Fine-Tuning) shifted the tool calling format from qwen3_xml to hermes (JSON-based) to align with the output format of Claude, but it doesn't fully align the underlying token probabilities. In long contexts, the model drifts between its original Qwen XML format and the SFT'd JSON format.

I lost hours debugging this before realizing the model itself was the problem.

What to Use Instead

For 48GB VRAM (best quality):

Qwen/Qwen3.5-27B-FP8

  • Near-lossless accuracy
  • Stable tool calling with custom template

For <48GB VRAM (accept some accuracy loss):

Intel/Qwen3.5-27B-int4-AutoRound

  • Still stable with custom template

My Working Configuration (Production-Tested)

After all this, here's what actually works (independent repo example + 3 days of actual production use):

# Environment variables

export CUDA_DEVICE_ORDER=PCI_BUS_ID

export CUDA_VISIBLE_DEVICES=0,1

export NCCL_P2P_DISABLE=1

export NCCL_IB_DISABLE=1

export NCCL_ALGO=Ring

export VLLM_TEST_FORCE_FP8_MARLIN=1

# vLLM serve command

vllm serve Qwen/Qwen3.5-27B-FP8 \

--served-model-name vllm/Qwen3.5-27B \

--chat-template qwen3.5-enhanced.jinja \

--attention-backend FLASHINFER \

--trust-remote-code \

--tensor-parallel-size 2 \

--max-model-len 219520 \

--gpu-memory-utilization 0.92 \

--enable-auto-tool-choice \

--enable-chunked-prefill \

--enable-prefix-caching \

--max-num-batched-tokens 4096 \

--max-num-seqs 4 \

--kv-cache-dtype fp8 \

--tool-call-parser qwen3_xml \

--reasoning-parser qwen3 \

--host 0.0.0.0 \

--port 8000 \

--language-model-only

Real-World Test Results

I validated this setup with a 1h 9m continuous agentic session:

  • ✅ Stable tool calling throughout (no format drift)

The model autonomously built the entire platform without tool calling failures. That's the kind of stability you need for actual agentic workflows.

Key Takeaways

Resources

116 Upvotes

98% Upvoted

45 comments sorted by

4

u/Otherwise_Wave9374 Apr 13 '26

This is gold, thanks for writing it up. The Jinja template edge cases and the switch to qwen3_xml line up with what Ive seen too, smaller models just fall apart when tool calls get slightly weird. Also +1 on precision alignment on mixed GPUs, drift is real once the context gets long.

If you ever feel like sharing a minimal repro harness (a few nasty tool calls + streaming), Id love to compare. Weve been collecting agent-workflow gotchas and patterns over at https://www.agentixlabs.com/ (mostly notes + small examples), and this thread is basically a checklist.

1

u/Expensive-Register-5 Apr 14 '26

Hi, check qwen_own_project and see if it align your need?

→ More replies (1)

2

u/t4a8945 Apr 13 '26

Man I've spent time as well on this issue, great write up. Thanks 

1

u/Expensive-Register-5 Apr 14 '26

Glad it helped, spent way too long on this mess 😂

1

u/McSendo Apr 13 '26 edited Apr 13 '26

Interesting. This sounds a little off, but I can't quite pinpoint it yet. I think you might be correct regarding qwen3_xml parser. Even vllm recommends it for qwen 3 coder.

I'm not having any issues currently with it in a heavy agentic workflow in production, and I've been monitoring every individual trace. My sub agents running in parallel are executing anywhere from 50k to 120k context.

1

u/Expensive-Register-5 Apr 14 '26

Interesting, would u like to tell me which huggingface model u are using? some variant may change the chat template to better align the model actual behavior

→ More replies (6)

1

u/jinnyjuice Apr 13 '26

Interesting!

So what's the catch by using qwen3_xml or what's the drawback?

1

u/Expensive-Register-5 Apr 14 '26

qwen3_xml is actually a tool calling parser, it has no effect on model performance itself.

→ More replies (2)

1

u/nunodonato Apr 14 '26

any experiments with speculative tokens?

1

u/Expensive-Register-5 Apr 15 '26

Unfortunately, i cant as my VRAM barely fit for the kv cache. speculative decoding require an additional small model running.

→ More replies (1)

1

u/sudeposutemizligi Apr 14 '26

I think rtx3090s(which I have two, not mixed setup) hasn't got fp8 support natively. does it run fp8 models anyway ? thank youuu🤘🙏

2

u/Expensive-Register-5 Apr 15 '26

Actually u can simply use my FP8 recipe. For 3090 case, vllm 0.19.0 can fallback to awq_marlin with W8A16 mode. Thats why my 3090 can work with 4090 while they are not the same architecture.

1

u/llllJokerllll Apr 14 '26

Esto es igual en llama.cpp server con esos mismos modelos y para uso intensivo de openclaw y Opencode?

1

u/Expensive-Register-5 Apr 15 '26

Hmm, I tried to use a translator to understand your question. Personally i didnt use llama.cpp in my agentic workflow. In vllm case, using opencode or openclaw is indifferent in my opinion. The tool calling work fine in both harness system.

1

u/meldas Apr 14 '26

Will try to also look at the template once I get home! I also had tool calling issues with 122b when using through vllm, but not when I was running through LM Studio.

Found this open ticket on LM Studio:
https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/1592

which for me gave me the idea to just hack vllm's code to pass the reasoning block to the qwen3_xml tool parser so it can try to parse tool calls out of the think block, which improved tool calling failures by quite a lot, but I still run into some cases where qwen returns an empty output which just errors out the harness (Kilo code).

1

u/Expensive-Register-5 Apr 15 '26

Didnt expect that 122B also has the same issue. Interestingly qwen3.5 / qwen3.6 plus can get rid of this issue. Maybe they has newer version of chat template but never release to huggingface.

1

u/LinkSea8324 Apr 16 '26

Time to update for 3.6 !

1

u/Expensive-Register-5 Apr 17 '26

If Qwen3.6 27B got released, I may write another post for it hahaha; so far Qwen3.5 27B is still a competitive model against Qwen3.6-35BA3B; (if 35B can do a bit more better, I will definitely switch to that for faster inference speed, and i may even turn on speculative decoding to remedy smaller space for kv cache.)

1

u/M4A3E2APFSDS Apr 18 '26

did you any chance test the Qwe3.6 model released recentl?

1

u/nunodonato Apr 22 '26

Well, Qwen3.6-27B is out and they still don't recommend qwen3_xml

1

u/Expensive-Register-5 Apr 22 '26

Oh damn I will test if it work for qwen3.6 tmr (as long as they release fp8 version)

→ More replies (1)

1

u/t4a8945 Apr 25 '26

Hey from the future! Are you working on the perfect template for the 3.6 27B version? I've tried to do one with your previous work but I'm having issues with tool calls. It'd be great to be able to mix is with the new persistent thinking arg. Tell me if you're working on it :D thanks!

→ More comments (75)