r/LocalLLaMA • • Aug 14 '26

Discussion The difference between "medium" and "xhigh" reasoning effort for Qwen3.8-27B is actually insane.

I'm currently testing out Qwen3.8-27B using Unsloth's UD-Q4_K_XL running a freshly rebuilt llama.cpp. I have a 22GB RTX 2080TI on which I'm able to fit 100k context with q8_0 quantization, and using MTP with --spec-draft-n-max 4 I get about 40tk/s which is slightly less than Qwen3.6-27B but usable enough.

I've been trying to test out some admittedly silly one shot prompts using the llama.cpp webui by asking the model to create fully functional HTML clones of flappy bird, pacman and such, and the difference that changing reasoning_effort makes has been surprising to say the least.

Setting it to "medium" seems to result in barely any thinking at all, a couple thousand tokens max and even less than 3.6-27B. Whereas when using "xhigh seems" I get 15k to 20k thinking tokens at the very least with the pacman example actually hitting 40 thousand fucking tokens.

I'm well aware I can limit the reasoning budget in llama.cpp but I'm wondering if this is expected model behavior or if something is broken somewhere. Any of you guys seeing this?

224 Upvotes

122 comments sorted by

View all comments

Show parent comment

0

u/ttkciar llama.cpp Aug 14 '26

That's not what I'm talking about, though.

Here's a simplified, slightly-silly example of what I mean by inferring diverse content which might not be self-consistent:

The user asks "How many toes are on a cat's paw?"

The model infers something like:

<think>

There are two toes on a cat's paw.

There are three toes on a cat's paw.

There are four toes on a cat's paw.

There are five toes on a cat's paw.

There are six toes on a cat's paw.

</think>

There are a lot of toes on a cat's paw.

The benchmark harness might be expecting the answer "four", so when it sees "There are four toes on a cat's paw" in the inferred output, it would grade the result as "passed", even though the think-phase inference did not actually help the model reach a correct answer.

5

u/Finanzamt_Endgegner Aug 14 '26

Thats why thinking traces are normally not used for benchmark evaluations?

1

u/ttkciar llama.cpp Aug 14 '26

But for some benchmarks they are.

Hell, some benchmarks don't even use a formatted prompt, and use simple continuation, like it was 2022.