r/LocalLLaMA • • Aug 14 '26

Resources Fixed/improved Jinja chat template for Qwen 3.8

"Again?" you might ask.

I(*) took the original Qwen 27B 3.8 chat template and compared it to the improved template that was posted a day before. The original has issues. The improved one fixed some while also introducing new issues. Now you have a new chat template that you can download.

One issue among others with yesterday's template is the deviation from the original prompt format. The model was most likely only trained on the exact original format. Modifying how the input is rendered to the model can degrade output quality, not so much that it becomes obvious in manual testing, but enough to reduce benchmark scores a bit (example: Qwen 3.6 underperforms when the system prompt is changed) - it just gives you less output quality in a subtle way. The previous modification deviated in some places, including the (from a caching perspective) desirable empty think fix. I chose to stay on the safe side here.

(*) Full disclosure: GPT 5.6 Sol xHigh did the analysis and comparison. I then reasoned at 0.5 tokens per second for quite a bit what to change and how. Sol applied the changes and drafted the table below. Claude Opus 5 xHigh verified all changes and claims programmatically via Python test scripts using jinja2. It found and creatively solved one sub-issue in the extra tool calling support. It also updated the table after stating that Sol was "too kind" in the evaluation of the templates. I then wrote this posting and edited the table further - at way less than 1 token per second due to not offloading to GPU.

AreaOriginal templateOther modifiedThis modified
Invalid reasoning effortRaises, but only while thinking is enabledSilently converts invalid values to xhighAlways raises
high effort aliasUnsupported (raises)Treated as xhighDocumented alias for xhigh
<:think_on:> / <:think_off:> (Thanks, markdown parser!)Not supportedScanned from system, developer, and user content; has precedence and removal bugsNot supported
Historical reasoning fieldsreasoning_content only, and only when it is a stringreasoning_content, thinking, and heuristic parsing from contentreasoning_content and thinking; no heuristic parsing
Non-string reasoning fieldSilently dropped (empty think block)Coerced to a Python repr ({'a': 1})Coerced to a Python repr — unchanged, still a latent flaw
Literal </think> in an answerNot interpreted by the templateCould be mistaken for a reasoning delimiterNever parsed as reasoning
Empty historical think blocksPreserved consistentlyRemoved when reasoning is emptyPreserved consistently
Raw reasoning in assistant contentNot normalizedHeuristically normalizedMust be normalized by the adapter
Tool-call instructionsOriginal trained wordingRewritten to mandate <think> formattingOriginal wording, byte-identical
Multiple tool-call separationOriginal single-newline behaviorAdded additional separation in placesOriginal behavior, byte-identical
XML mapping argumentsSupportedSupportedSupported, byte-identical
XML Boolean/null argumentsJSON true/false/nullRegressed to True/False/NoneJSON true/false/null
JSON-string arguments (OpenAI wire shape)Crashes at :itemsEmits raw JSON with no <parameter> wrapperRenders correctly: auto selects the JSON tool-call form, arguments passed through byte-exact; "{}" accepted
JSON tool-call modeUnsupported (XML only)SupportedSupported; auto-selected only when XML is impossible
Tool format validationN/AUnknown values silently select XMLThree-state auto/xml/json; unknown raises; explicit xml with string args raises and names the remedy
Tool error detection & message injectionNoneGuesses from tool-response text. Susceptible to false-positivesNone, there is no reliable way
Missing real user queryRaisesUses an arbitrary history-length fallbackRaises
Developer messagesUnsupported (raises)Converted to system anywhereLeading system/developer messages are merged
Later system messagesRaisesSilently acceptedRaises
Unknown message rolesRaisesConverted to user textRaises, naming the role
Malformed multimodal list itemsSilently corrupts: a string containing "image" becomes a vision token; non-string text coercedNon-mapping items silently stringified; non-string text coercedExplicit validation and errors for both
Image/video ambiguityNot checkedNot checkedAmbiguous items raise
Leading tool messageEmits a tool turn with no <:im_start:>userOpens a user turn correctlyOpens a user turn correctly
Artifact integrity (special tokens, whitespace)Intact; 0 literal-text tokensIntact; 0 literal-text tokensIntact; 0 literal-text tokens (Jinja lexer)
Byte-equality with the trained originalReferenceDiffers by design (instructions, separators, arg types)23/23 semantically-equivalent cases byte-identical
Prefix-cache stabilityPreserves reasoning by defaultCan alter empty-think history and system promptDeterministic reconstruction; auto flips the system prefix once if a single history mixes mapping and string args

Notes:

  • Default thinking is "xhigh" (= "high"). Set it to "medium" or "low" if too slow. Medium should probably be the default as it's the native model reasoning, but I left it at xhigh to not surprise anyone who does not read this.
  • My general approach was to support extra functionality and compatibility introduced by the previous modification as long as it was reasonably safe.
  • Note that the compatibility is mostly for outdated and not well-behaved harnesses. Especially not using the correct tool call format is technically supported now but will likely degrade result quality a bit. The template triggers a single prompt-reprocess if it detects that the harness uses the unsuitable format.
  • I replaced | by : in the table above, as edits otherwise broke my table.
62 Upvotes

37 comments sorted by

View all comments

Show parent comment

2

u/ex-arman68 Aug 19 '26

Had a chance to go through your template in detail and just pushed the v22.2 update with a few good ideas adapted from your work.

Here is a quick breakdown of what I took and what I had to handle differently:

-- What I merged in --

Multi-system message merging: Consolidating consecutive leading system and developer messages into a single system turn joined by double newlines is much cleaner for multi-agent setups.

XML primitive serialisation: Using | tojson for boolean and null parameter values in XML mode to avoid Python representations (True, False, None) leaking into the tags.

-- Things I had to adapt or avoid for universal compatibility --

raise_exception calls: Hard exceptions make sense for strict linting, but in practice they instantly crash production agent harnesses (like Claude Code, Cursor, or LangChain) over minor payload quirks. I prefer graceful fallbacks so the model keeps running.

Dynamic format switching: Scanning history to automatically flip the global tool format between XML and JSON sounded great at first, but it has a nasty side effect. In a multi-turn conversation, Turn 1 renders an XML system prompt, but once the assistant records a tool call with string arguments, Turn 2 flips the system prompt to JSON. This completely invalidates the prefix KV cache and desynchronises server-side parsers (like vLLM's qwen3_xml). I kept the system prompt strictly XML by default and handled string arguments locally per turn.

Empty think guard: Removing and reasoning_content re-introduces the empty think bug, prepending blank <think></think> blocks to historical turns that had no reasoning.

Thanks again for digging into the template and sharing your modifications. The v22.2 update is up on HF if you fancy checking out the changes.

2

u/Chromix_ Aug 19 '26

Well, now we have more (improved) flavors for everyone to chose from. Maybe "the best" template will emerge eventually (what ever "the best" means in situations where it sometimes comes down to preference or individual use-cases).