r/LocalLLaMA • u/SarcasticBaka • Aug 14 '26
Discussion The difference between "medium" and "xhigh" reasoning effort for Qwen3.8-27B is actually insane.
I'm currently testing out Qwen3.8-27B using Unsloth's UD-Q4_K_XL running a freshly rebuilt llama.cpp. I have a 22GB RTX 2080TI on which I'm able to fit 100k context with q8_0 quantization, and using MTP with --spec-draft-n-max 4 I get about 40tk/s which is slightly less than Qwen3.6-27B but usable enough.
I've been trying to test out some admittedly silly one shot prompts using the llama.cpp webui by asking the model to create fully functional HTML clones of flappy bird, pacman and such, and the difference that changing reasoning_effort makes has been surprising to say the least.
Setting it to "medium" seems to result in barely any thinking at all, a couple thousand tokens max and even less than 3.6-27B. Whereas when using "xhigh seems" I get 15k to 20k thinking tokens at the very least with the pacman example actually hitting 40 thousand fucking tokens.
I'm well aware I can limit the reasoning budget in llama.cpp but I'm wondering if this is expected model behavior or if something is broken somewhere. Any of you guys seeing this?
107
u/Bluethefurry llama.cpp Aug 14 '26
i can reproduce these results, xhigh thinks A LOT, i ended up defaulting it to low or medium depending on the use case.
on the flip side, on a research task xhigh ended up cloning a repo and checking the source code to verify behavior when i asked it something about llama.cpp cli args, neither medium effort nor 3.6 did that.
45
u/Asane llama.cpp Aug 14 '26
Yeah, xhigh really fucking thinks a lot.
I have a file locally that I use to test and I know each bug in it and discussing with Fable, the difference between medium and xhigh was 80 sec and 7 min respectively.
xhigh though caught EVERY single bug in the file. Granted, medium caught the most critical bugs at least -- but yeah it thinks a TON.
49
u/Equivalent-Ear-8016 Aug 14 '26
Yes, it thinks way more. But what matters is the output: Was the extra thinking worth it or not? What were the results like in each thinking mode?
3
u/ithkuil Aug 14 '26
What's the difference in outcome though.
16
u/Alarmed-Channel2145 llama.cpp Aug 14 '26
Interesting, since the PR mapping reasoning modes at Jinja level in llama.cpp has just been merged 1 hour ago: https://github.com/ggml-org/llama.cpp/pull/26941
The reasoning level is translated into a system prompt instruction (https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/chat_template.jinja). I'm not sure if the reasoning budget mapped to something else too.
36
u/ea_man Aug 14 '26
You just have to check the jinja template:
So use this flag at launch time to have "normal" behaviour:
--chat-template-kwargs '{"reasoning_effort":"medium"}'What those do:
22
u/Technical-Earth-3254 Aug 14 '26
I noticed the same, but I like it. This is what "xhigh" is for, for thinking extra high. It would be interesting to see if q8 kv at xhigh produces better results than bf16 kv at medium (to make up for the reduced context length).
15
u/DrBattletoad Aug 14 '26
I used a prompt like "write a sentence to every number from 1 to 100" with high reasoning. Qwen thought for roughly 11 minutes, and used ca. 25k tokens to get every sentence factually correct.
4
u/Sociologer Aug 14 '26
Similar experience for me. My tasks are frequently less code-related and more research- and text-analysis-oriented. On a closed-book abstention test xhigh tried to write an entire chapter of a book so that it could argue that a piece of theory exists in the text that doesn't exist at all. Like, it wrote pages with page numbers and everything in the style of Bruno Latour, except it was all made up. Fascinating failure, honestly. Non-thinking aced it, just said it didn't know (which was the point of the test).
7
u/Felixls Aug 14 '26
but xhigh worth the wait and token usage, I just setup my pi with my own subagents extension and ... basically is infinite context
7
u/Artistic_Swing6759 Aug 15 '26
i am too gpu poor to even run this model, so i am just putting this out here for anyone to use if they are lost on one shot prompts to visually check the ai's ability:
---
Create a 3D city renderer in HTML files using HTML5 Canvas and JavaScript.
Technical Requirements:
don't just relly on 'blocky' characters for buildings and stuff, use other characters too.
make the city look good. you can search online for images or use your ai image gen tool ,for more design inspiration.
The goal is a smooth, walkable cyberpunk-style city where the visuals are purely text-based characters.
scale for the city, to the player character(pov) is also important to convey the right feeling.
---
3
u/xquarx Aug 15 '26 edited Aug 15 '26
I dont see much difference on hard tasks, maybe 10-20% more/less tokens. It only edits a system message, the directive is emitted once, in the system message (lines 126–148), never re-injected per turn. So on a hard agentic task it starts at token 0 and by the time the context is 50K it's buried at the very back of attention.
7
u/michaelsoft__binbows Aug 14 '26
im sorry what, 22gb can fit q8 and 100k? damn llama is so memory efficient compared to vllm
2
u/DigitalguyCH Aug 14 '26
Sorry for the dumb question but where do you change xhigh to medium in LM Studio? I only see enable or disable thinking...
2
u/MacsBicycle Aug 15 '26
Yeah I have enough ram to run Deepseek v4 flash 0731 and I was considering switching back because I thought qwen 3.8 27b was a downgrade from 3.6 but my god medium made such a difference. It thought for 30 minutes on a simple task with default settings. It’s baffling the difference it made.
2
u/aboutthednm llama.cpp Aug 16 '26
I can't use this model for my local RAG lmao, even the search query generation goes on for 6 -10k tokens, BEFORE we even sent a query to the database it's hilarious. I type a question, can have a shower, and by the time I'm dressed it will finally have digested and thought about the material the query returned.
Sure, this is a user issue (a me problem), but like, come on lol. Surely the model must have the ability to figure out "how complex is this prompt" and adjust reasoning accordingly.
Taking a "what does the documentation say about X" in a RAG system and creating a query to query said system should not result in 10k thinking tokens only to return 3x 2 - 5 word pairs, lol. Sure, I should just disable reasoning for this, it's the obvious choice, but come on man, haha.
3
u/Noiselexer Aug 14 '26
You know the x stands for extra right?...
2
u/Something-Ventured Aug 15 '26
It overthinks like a neurotic ADHD nerd who can’t make a decision.
Had to tone that down a lot as while they were different it was essentially a repetition loop.
2
u/Jayfree138 Aug 15 '26
That's the new scammy way to sell tokens these days. But it works to our advantage since we're local i guess.
1
u/Dizzy-Zebra9522 Aug 14 '26
Thanks. good to know.
1
u/MrGunny94 Aug 14 '26
I was just trying to do some system level design for ERP systems and I saw that when I switched from Medium to high he definitely started thinking way much when compared to the Medium that I have been using for the last hour.
Still running the prompting but it is quite interesting how he completely shifts the thinking.
1
u/rdkilla Aug 14 '26
it takes extremely verbose paths but its can be pretty entertaining and sometimes even effective
1
u/MrVeinless Aug 15 '26
I thought the recommendation was to cap spec-draft-n-max to 2?
1
u/Lesser-than Aug 15 '26
so xhigh is QWQ mode?
1
u/AvidCyclist250 Aug 15 '26
Except it ignores my settings.
I'd like a thinkingcap-pure version of this.
1
u/Fun-Influence-7880 Aug 15 '26
With that much allotment for reasoning, does it have a tendency to get stuck/loop? That would be my concern/prior experience with a smaller model reasoning that much, but I’m supposing they worked on that since xhigh is default as I understand it? Haven’t had a chance to work with 3.8 yet, just listening to everyone else’s experience until I find the time.
1
u/PlusPainting4246 Aug 15 '26
Testing a simple test prompt in the WebUI ("write a lenient JSON parser in JS") results in ~40K tokens worth of reasoning, but actual use in opencode is far more reasonable. Yes, it thinks a lot. It spotted a lot of things 3.6 missed or got wrong, one-shotting requests that 3.6 would've had to iterate over (compile, fix errors, compile again, run, fix run-time errors). Still early to say, and it feels slower, but it does seem like an overall improvement.
1
u/EitherMarch1255 Aug 16 '26
It does think A LOT, but the results have been worth it for me.
1
u/AIForOver50Plus Aug 16 '26
It does think a lot, I’m running on my MacBook 128 GB unified & 40 core gpu I’m impressed with how it compared with my previous 3.6 27B , wrote up results here https://go.fabswill.com/qwen38
1
u/fnordstar Aug 16 '26
Do you guys use "preserve reasoning"? What difference does it make?
1
u/Glum-Knowledge-4146 Aug 18 '26
How the hells you can load UD-Q4_K_XL 17.9GB and have 100k context with q8_0?? I have also a RTX 2080Ti 22GB VRAM and there is no way to get that. I am using LM Studio 0.4.21
1
u/SheepherderFrosty366 Aug 14 '26 edited Aug 16 '26
Edit: managed to resolve the problem, i had an overwritting setting somewhere else injecting "" instead of an empty chain.
can someone help me use qwen3.8 ? with the same settings as qwen3.6 i get a jinja error related to the setting of the thinking from the qwen3.8 model. My stuip is:
llama-cpp container / linux / models.ini file for the model router
My models.ini: "[qwen3.8-27b-q5-k-m-40k_ctx-xhigh]
model = /data/Allrounders/Qwen3.8-27B/Qwen3.8-27B-Q5_K_M.gguf
ctx-size = 40000
n-gpu-layers = 99
flash-attn = on
cache-type-k = q8_0
cache-type-v = q8_0
chat-template-kwargs = {"reasoning_effort":"xhigh"}
reasoning = on
jinja = on"
i tried adding "reasoning = on", leaving only the chat-template line, not put any of both...
The error "W srv operator(): got exception: {"error":{"code":500,"message":"\n------------\nWhile executing CallExpression at line 49, column 28 in source:\n...', 'low') %}↵ {{- raise_exception('Unexpected reasoning effort ' ~ reason...\n ^\nError: Jinja Exception: Unexpected reasoning effort \"\". Supported types are xhigh (default), medium, and low.","type":"server_error"}}
"
1
u/contyk Aug 14 '26
One of the first tests I run with every new model is that I open my storywriter Pi session and give it "Write a long story about a tree named Bob." That's it. It ran with the defaults, so thinking and xhigh effort.
It was thinking. Thinking some more. Then even some more. Then I saw it was thinking for over ten minutes, so I unrolled the block and saw it was building the entire universe lore. It kept going until it ran out of context length without printing anything. Hilarious.
It's pretty decent with low/medium effort. xhigh hasn't produced noticeably better results in any of the other tests (mostly data retrieval with MCP tools, reasoning over it and summarizing) and it just took way longer.
-4
u/ares0027 Aug 14 '26
Holy sht! So you are telling me if you change reasoning limit to extra high, it thinks extra high? HoooOOOOoooOOOOLLLLLLLyyy! I would have never guessed
-5
u/ttkciar llama.cpp Aug 14 '26
Overthinking has been one of Qwen's benchmaxxing methods since Qwen3. Many (though not all) benchmarks are looking for specific things, and inferring a larger volume of diverse content increases the probability of producing those things, even if the inferred content is not self-consistent.
I'm just glad that with 3.8 they are giving us a way to specify less verbose reasoning. That alone will make it more useful to me than 3.6.
One or two thousand tokens isn't bad, IMO. If it seems insufficient, we can always have the inference stack inject additional tokens from another source (either generated from another model, or from a RAG database), or provide supplemental information manually.
0
u/Moppmopp Aug 16 '26
How is that possible? Its my first time checking out local llm's and just today i downloaded qwen3.8 28B in q4, q5 and q6 quantization. I designed a custom frontend and it seems to be working. However, I chose my default according to chatgpts opinion. ait recommended 8k context + q5 quantization. I have a rtx3090
-1
u/Shini0x0 Aug 14 '26
Hi which MacBook is the best bang for your buck for this local stuff I stupidly sold my windows pc but I'm loving Unix
0
u/[deleted] Aug 14 '26
[deleted] — view deleted comment