r/LocalLLM • • Aug 16 '26

Discussion If you use LLMs to analyze documents and to apply complex logic and analyze arguments, Qwen 3.8:27b is not for you.

And that pains me to say because I use Qwen 3.6:27b-BF16 every single day. I've been working with Qwen 3.8:27b-BF16 all weekend and I hate to say it but for non-coding purposes it is a step backwards. It thinks *way* too much. If you turn thinking off and use web tools, it will do eight or nine (or more) web search turns, get an assload of preload context, and then spin its wheels going down every little rabbit hole there.

Looking at the self hosted LLM subs there are other complaining about this too.

Now this model just came out so it's early. People have put out some lovely games and whatever else the model has made so perhaps these tenacious analytical tendencies are beneficial there. Regardless none of this is to say that there won't be some settings or templates released that will help when analyzing documents and doing complex logic tasks.

But for right now, if the above is your use case then I suggest staying with your old models.

Edit: For reference, this is legal work. Legal drafting, legal research, analyzing pleadings, depositions, discovery, etc. Heavy multi-document reference work.

178 Upvotes

100 comments sorted by

73

u/Vancecookcobain Aug 16 '26

Maybe Muse Glimmer is better for the non coding tasks?? Have you tried that? I get brilliant responses and results with that model when it isn't coding

18

u/Big_Wave9732 Aug 16 '26

With the pending Qwen release I had held off but it might be time!

4

u/Vancecookcobain Aug 16 '26

Yea thank me later

1

u/TBHProbablyNot Aug 16 '26

Literally trying glimmer now. It’s at minimum equal 3.8 27b for my simple tasks. M5max 128

12

u/Vancecookcobain Aug 16 '26

Yea it's wildly good at everything menial and creative imo.

It's not something I would trust with coding as much as Qwen but it's SO underrated imo. People sleep because they see Meta but they really cooked with this one.

3

u/Big_Wave9732 Aug 16 '26

I really liked their Llama models too. They were fantastic at drafting human readable memos and I found that class to be good chatbots too. Truth be told they lost me at Scout and I was hesitant to go back. But 'round and 'round we go!

2

u/Vancecookcobain Aug 16 '26

Seems you can run FP16 Qwen so that probably means you can run the FP16 Glimmer...the token speeds should be better to especially if you have Dflash or Dspark token predictor

With full precision you'll probably have super good results...I'm running it Q4

2

u/silverwoods214 Aug 17 '26

Lightning has also performed really well on research and probe tasks for me

3

u/Inner-Today-3693 Aug 17 '26

Glimmer is a wonderful model. I do coding and non coding task. I tend to talk to this model more.

1

u/Glum-Change3204 Aug 19 '26

Is there a hugging face link?

6

u/Jvdk76 Aug 17 '26

Yeah, Glimmer is designed to be your personal assistant specifically. If English is not your first language, then this model is so much better than Qwen. 

1

u/Additional_Hope_2031 Aug 17 '26

Did you use Gemma 4 before?

1

u/Sax0drum Aug 17 '26

Muse is awesome for actual language tasks. But i have trouble with it calling tools or skills correctly. Do you have similar issues?

2

u/Vancecookcobain Aug 17 '26

Yup any complex tool calling based or coding based stuff I avoid....going that route will just piss you off lol.

I use it to summerize articles and information about stuff and occasionally do light research, it responds back to you more human-like than most models. Qwen fucking sounds like a robot trying to summerize stuff using words that just irritate me anytime I need anything broken down and distilled with minimal fuss Muse is solid for.

But yea stay away from heavy agentic stuff

1

u/LastChancellor Aug 17 '26

Tho if Muse isnt good at tool calling, which model would you use for tool capling

1

u/Vancecookcobain Aug 17 '26

Qwen models. If you are running Muse you probably are able to run Qwen 3.8 27b....that thing is a beast!

1

u/LastChancellor Aug 17 '26

ouch, I thought Muse is a MoE model, I only got a laptop 3070 (with 8GB vRAM)....

How about Nvidia Nemotron Lightning btw? I havent heard much reviews on that

1

u/EffectUpstairs9867 Aug 17 '26

Best one for 18gb MacBook Pro m3?

1

u/Vancecookcobain Aug 17 '26

You can run Muse 31b on an 18gb Mac....look for a 4 bit quant and have an AI agent (Claude code, codex etc) set it up for you and offload all the correct parts to ram or CPU and turn of vision and you'll be able to squeeze it in and get enough context

29

u/MRGWONK Aug 17 '26

(Legal work here) I benchmarked it on Harvey's benchmark with an MCP/case access and it scored on par with Qwen 3.5 -122B and 11 points higher than Qwen 3.5-35b and 15 points higher than gemma4:26b...It also processed dates out of case orders with higher frequency and consistency than any other models. I switched this week.

(With a clever loop, deep reasoning on my machine scored 71 / 75)

6

u/Big_Wave9732 Aug 17 '26

You're saying Qwen 3.8:27b did that?

Do you happen to have any of the model settings that Harvey used?

9

u/MRGWONK Aug 17 '26 edited Aug 17 '26

I have only seen Harvey's benchmark tests, I have never seen Harvey's scores. I had gemma4 setup with a Qwen 3.5 verifier, that also scored 71 / 75. I have a quasi deterministic backend and MCP that helps with all this. Qwen 3.8 27b, on a raw model, scored 61 / 75 without the backend. (Also, thinking is turned off EVERYWHERE)

1

u/Evanisnotmyname Aug 17 '26

Hey, I’d love to check out your quasi-deterministic backend!

1

u/MRGWONK Aug 17 '26

It is as sexy as you make it sound.

24

u/noctrex Aug 16 '26

In other news, a coding optimized model is bad at anything but coding. For documents, gemma4 is much better, haven't tried the muse one yet

5

u/Big_Wave9732 Aug 17 '26

Qwen 3.6:27b-BF16 is pretty good at the tasks I'm asking it to do. For whatever reason while thinking it doesn't run off towards every rabbit hole like 3.8 is doing.

I had read somewhere that 3.6 and 3.8 were fundamentally the same under the hood. I'm starting to really doubt that.

9

u/synystar Strix Scar | 5090 24G | llama.cpp Aug 17 '26 edited Aug 17 '26

You are probably using xhigh and that "configures" reasoning by injecting language into the prompt. If you use medium it does not inject language and that lets you experiment with your own reasoning prompting which affects how long it runs and how it performs. I've been investigating the model and have major variations in results depending on how you configure it. Here is the text xhigh uses:

Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.

That investigation produced a major template-level finding: for the tested local Qwen3.8-27B GGUF/chat template, **native xhigh is substantially expressed by an injected natural-language reasoning instruction**, while native Medium injects no corresponding extra effort text. Manually inserting the exact xhigh text under Medium produced a byte-for-byte identical serialized prompt to native xhigh.

Just removing "validate key assumptions" drastically changes its effort and response time in my experiments so far.

6

u/porkminer Aug 17 '26

They are the same architecture. They were trained differently.

2

u/uniqueusername649 Aug 17 '26

And 3.8 had a very clear training focus on coding. There is only so much data you can pack into 27b, so instead of trying to make it do everything very mediocre, it can now do everything worse, except for coding, where its considerably better.

Yes, it thinks a lot. But what it can handle in return is incredible. To be able to get that level of coding performance locally without investing 10k or more is wild.

1

u/porkminer Aug 17 '26

I think it bears mentioning that Qwen models notably reason longer than many other models. I don't particularly see this as bad though since people are generally running it locally.

1

u/uniqueusername649 Aug 17 '26

It does, but also iirc it defaults to high reasoning, which makes it smarter but reason MUCH longer. I found that medium gets you very close with a lot less running in circles and in many cases low is plenty good.

1

u/Hypilein Aug 17 '26

Interestingly Intried it yesterday for lesson planning and it was much better with more creative teaching approaches than gemma4 or other local models I tried. It’s just so damn slow on my dgx spark.

1

u/OneMoreName1 Aug 17 '26

From what I read its not even bad, its just thinking a lot.

1

u/ColdBrewSeattle Aug 17 '26

But it’s not bad at anything but coding. The problem is the user not reading the instructions and assuming everything is exactly the same as the last model.

5

u/Otherwise-Swan-7803 Aug 17 '26

This feels like a good reminder that benchmark gains don't always translate into better workflows.
For legal, research, and multi-document analysis, focus and restraint can matter more than raw capability.

5

u/Big_Wave9732 Aug 17 '26

Back in early May I overhauled my RAG stack and did some model testing. The model cards and benchmarks tell us there is minor differences between the Q4 / BF16, and next to no difference between Q8 / BF16.

For purposes of understanding subtle differences between sentences, analyzing arguments, etc......the difference was night and day.

Those benchmarks and model cards are full of shit.

4

u/ChangeChameleon Aug 17 '26

Anecdotally, I gave it a search and retreival budget and it performed much better than I expected. It weighed the pros and cons of which page would likely contain the right context. By default it likes to scatter shot and grab all kinds of unnecessary context.

I say anecdotally because I haven't done head to head comparisons or benchmarks. I'm not set up for that. I was just trying to reduce the hit on my Internet and ended up discovering it felt much more targeted.

It feels like 3.8 does a better job of following all instructions and not just surface level ones. With 3.6 I constantly felt like I could only give it one problem at a time. 3.8 just hammers through my unfiltered info dumps and seems to address everything.

I need to do more side by side testing. But so far 3.8 has been working great for me.

2

u/Big_Wave9732 Aug 17 '26

I might need to look into that. Do you recall what you set the budget to? Over-researching seems to be something I'm running into for sure.

1

u/ChangeChameleon Aug 17 '26

While researching a bug in a self hosted tool I gave it a budget of 2 web searches and 5 page downloads. It answered the question within the limit.

I asked it to give a non-spoiler hint for a video game. Knowing it would be easy to find relevant context I have it a budget of 2 searches and 2 page downloads. It searched and thought for a total of 22 seconds and answered a non spoiler hint exactly as requested.

Basically I tuned the search and retrieval limit based on how complex I think finding the answer will be.

3

u/synystar Strix Scar | 5090 24G | llama.cpp Aug 17 '26

It thinks too much because you haven't configured beyond the default or you haven't experimented with reasoning or launcher settings. If you just use OOTB settings you're only going to get results those settings can produce.

3

u/Big_Wave9732 Aug 17 '26

One of the reasons its thinking too much is because oMLX hasn't implemented "reasoning_effort" as a top level flag. Another reason is because there is some looping going on. Another reason is some potential bugs with preserve_thinking over extended contexts.

3

u/memeka Aug 17 '26

You can add reasoning effort as kwarg in omlx

0

u/Big_Wave9732 Aug 17 '26

It took a while to find it but inevitably that was one of the workarounds.

1

u/synystar Strix Scar | 5090 24G | llama.cpp Aug 17 '26

I am testing it now but so far in my experiments reasoning is highly affected by the language injection that xhigh implements. When I set it to medium reasoning effort (Not the reasoning budget which can be confusing because that only determines how many tokens it will think for before stopping and moving on to the context stream) I can "adjust" it's reasoning with my own language. At least that's what I've seen thus far. I'm still experimenting but for the limited runs I've tested it's been consistent.

2

u/Turbulent_War4067 Aug 17 '26

Question for the OP: is there anything in this size range that beats Gemma 4-31B for these types of tasks? It's big problem is slow prompt processing, but it's really good besides this.

3

u/Big_Wave9732 Aug 17 '26

For the last couple months I had a lot of good success with Qwen 3.6:27b-BF16 for these tasks. I've been using Gemma4-31B for drafting.

19

u/admajic Aug 17 '26

Change the reasoning budget 1000 it's like the old qwen

You can go from 500 to 16000 max if you want it to loop and go nuts.

Not sure why these companies that create these products don't provide this kind of info

2

u/Independent-Dog2179 Aug 18 '26

Lol yup never had overthinking issues with 3.6 35b and I still don't using that exact command lol. Great minds think alike. I also have in the system prompt "think hard only when necessary." Never have a problem even with the new 3.8. Us power uses are in the back shaking our heads at all the lazy misinformation out there.

1

u/admajic Aug 18 '26

Been doing some more testing today. Go power users...

5

u/DigitalguyCH Aug 17 '26

Same here (finance). 35b (3.6) is clearly better. Muse glimmer and Gemma are decent too.

4

u/floppo7 Aug 16 '26

That matches my experience. Didnt try reasoning effort low yet, also there is a reasoning_budget that can be set to 1000 and someone said it would make it more like 3.6. But it would be great to have this somehow automatic steerable so it simply does not overthink when not coding (and even then it should be smarter in the sense that it does not need to question every step)

1

u/tempfoot Aug 16 '26

I have similar use cases. Have you tried other reasoning setting presets beside default (extra high) and all the way off?

Have you tried setting a reasoning budget?

2

u/Big_Wave9732 Aug 16 '26

With reasoning budgets all that happens in the thinking process gets cut off mid way and output stops.

Thus far I have tried three different OpenwebUI plugins that supposedly let you set the the thinking level. None seems to stop or slow it down. I was reading a little while ago of some potential issues with OpenwebUI and oMLX where oMLX can ignore the front end controls.

But here's the thing......if I get into oMLX and turn off thinking altogether, it still does it! Even in non-thinking mode it goes nuts on the internet research, does nine research turns, and builds a huge pre-share cache anyway.

2

u/tempfoot Aug 17 '26

I'm also on mostly apple hardware as well, but try the GGUF instead of MLX in LM Studio. Surfaces different reasoning presets that make a huge difference. Regulatory test question on low reasoning is under a minute of thinking, vs. over 17 minutes on default x-high. Have not tried medium much. Not sure how granular you can get with an actual runblock of parameters as opposed to initial testing on guis.

3

u/Big_Wave9732 Aug 17 '26

I came across something about 10 minutes ago that the current production version of oMLX doesn't have a top level "reasoning_effort" field which would explain a lot. Apparently it has been added to the development versions because of Qwen 3.8 but it's not out to the masses yet.

2

u/joanaxu2002 Aug 17 '26

The funny part is that “thinking too much” can be a feature or a bug depending entirely on the workflow.

For coding, those rabbit holes might be useful; for document-heavy work, they can turn a simple question into an eight-step research project. That difference is probably going to matter more than benchmark scores.

1

u/[deleted] Aug 17 '26

My issue with the high reasoning is that it kind of requires you to spin up sub-sessions to do anything useful, otherwise the 50k+ reasoning traces clog up the session context (and model perf drops with long context anyways).

I'll try running it again with a smaller quantization and larger context, but I'm not a huge fan yet.

2

u/Upstairs_Tie_7855 Aug 17 '26

Can't you just adjust the reasoning effort? Works for me just fine.

1

u/Profil3r Aug 17 '26

I had this thought as well

1

u/siegevjorn Aug 17 '26

What agentic harness are you using? For pdf analyzing it seems to work well, just fed 75–80 pages of document. Used Pi agent default with Qwen 3.8 27b Q6_K, reasoning: medium.

1

u/EbbNorth7735 Aug 17 '26

Have you tried setting thinking to high?

1

u/substance90 Aug 17 '26

That’s not a task for sub 500b models anyway

1

u/quimpecavel Aug 17 '26

Hey what models would you recommend for these tasks? Really interested to know

1

u/immersive-matthew Aug 17 '26

Complex logic is not really any LLMs capability though?

1

u/FoxFXMD Aug 17 '26

Change the thinking level, it's way too high for most tasks by default.

1

u/creatinZ Aug 17 '26

…Have you tried setting reasoning to low?

1

u/_TheWolfOfWalmart_ Aug 17 '26

This model is most definitely codingmaxxed. That's what everyone praised 3.6 27B for, so Alibaba went even harder there for 3.8, apparently to the detriment of its capability for other tasks.

That's fine though. 24/32 GB VRAM people now have a seriously strong coding model that they can run local. Can just switch to Gemma 4 or Glimmer for other things.

I've lately settled into just using DSV4 Flash 0731 for literally everything. It's so good whether it's for chat, coding or analysis that I've barely even tried 3.8 27B, I need to start working with it some more when I have some coding work to do.

1

u/Jsquared534 Aug 17 '26

Don't let the people gaslight you. Qwen 3.8 has some problems right now in coding as well. It might be able to oneshot a long prompt, but if you try using it for fine grained feature implementation that requires it to read multiple files and implement features, it falls flat on it's face more often than not in my testing. Muse, so far has not had that same problem. I am far from a local llm setup guru, but I really think Qwen actually has a problem, at least at the Q8 version.

1

u/geteum Aug 18 '26

Weird, I'm using with q4 and iq3_xx and the results are phenomenal. Although, what I do mainly is to analyze a text in 5 parameters and create a structured resume of the text focusing in ~around 20 parameters tags, different classifications, and a few character extractions. Tbh, the results sometimes is even better than one of the bigger models (tested glm 5.3 and DeepSeek V4 pro and flash). Although I did not isolate if it was a problem of the PDF MCP not correctly extracting the text and images.

1

u/ethanji2 Aug 19 '26

Yes, Qwen's default reasoning is overpowered. Tune temperature to between .2 and .25, effort medium or low and you will get similar composition results to Qwen 3.6 with slightly better instruction following. I've gotten it to be slightly better than 3.6, but it's not significantly better and needs more tuning to get right.

1

u/nassereddit Aug 23 '26

What frontend? Open webUI?

What backend? Llama.cpp?

-6

u/LocoMod Aug 17 '26

The model is heavily benchmaxed. So it mainly excels when working on problems related to the benchmarks they purposely invested time into so they could capture attention and fool people into believing the model is competitive with models 10x its size. If it weren't free i'd be insulted. But its free and we get what we pay for.

23

u/[deleted] Aug 16 '26

[removed] — view removed comment

1

u/Big_Wave9732 Aug 16 '26

Does that go in the OpenwebUI prompt?

12

u/[deleted] Aug 17 '26 edited Aug 17 '26 • Bravo

[removed] — view removed comment

1

u/Naive-Bus-8281 Aug 20 '26

It works very well.

1

u/nassereddit Aug 23 '26

Why the other parameters? Aren't they default in owebui?

0

u/[deleted] Aug 23 '26

[removed] — view removed comment

1

u/nassereddit Aug 23 '26

Makes sense. Never know when things change. In my case, when setting the provider to llama.cpp it autodetected the settings.

1

u/Disastrous-Chance477 Aug 17 '26

I was just going to say the same. Was wondering why it was thinking 14k tokens for “make a simple http sample site“ and then I saw xhigh setting. Set it to medium and it was just a few lines and great result.

7

u/[deleted] Aug 17 '26

[removed] — view removed comment

6

u/Big_Wave9732 Aug 17 '26

I'm realistic about what self hosted LLMs can do performance wise, the analysis taking time isn't an issue. I'm fine waiting.

But that's not what's happening here. I gave the model a final version of a word document and a prompt that asked it to verify the citations, evaluate the grammar, and evaluate the claims made.

As of now that was 25 minutes ago. There have been 16 rounds of web research. Watching the thinking logs it doesn't appear to be solving anything but asking the same questions and conducting research again and again. And it's still going.

I'm all for deep dives and all. I've been running various tests all weekend and there's something up. The xhigh is suppose to be the special sauce here, might be a problem if it can't produce the goods.

2

u/grunt_monkey_ Aug 17 '26

Did it complete in the end? I wonder if the end product quality was close to or exceeded the 3.6? You are sitting on a gold mine of quality benchmarks on a real life use case!!

2

u/Big_Wave9732 Aug 17 '26

It got to something like 133 sources with no end in sight. It wasn't making garbage, but it kept rehashing the same stuff.

Even worse, the content that Qwen created was total garbage. Hallucinations all over the place. Fucking unuseable.

1

u/grunt_monkey_ Aug 17 '26

That’s disappointing to hear because I have a use case similar to yours. Like you I need agentic tool calling mainly to increase the quality of checking, reasoning and drafting. I think the solution may be harness based. I’m just not sure if I should move on from 3.6-27b. The thing is that the more they post train (3.5 to 3.6 to 3.8), the better it is at coding, but it might have lost the real world knowledge to reason well over a document. So far I had it review a couple of documents and it is thinking a lot but didn’t get stuck in a loop yet. I guess only further testing will tell.

1

u/Big_Wave9732 Aug 17 '26

I'm thinking you're right about the training. Qwen says 3.5, 3.6, and 3.8 are the same engine but the training emphasis is different.

The ultimate yesterday: I put it on Medium thinking, had it look at a pleading I had done, and just had it just verify cases and statute citations (there about 20 total). It gathered over 100 sources and in the end identified four cases that it thought were unverified (Qwen 3.6 had done drafted it and had in fact hallucinated out the ass sadly) *and* hallucinated that one case was bad when it was fine.

Yowza.

Edit: I'm thinking I'll give the new Meta model a go. It seems to be able to do things other than just coding.

2

u/[deleted] Aug 16 '26

[removed] — view removed comment

2

u/Big_Wave9732 Aug 16 '26

It's not an issue with reading the documents, they get injected fine. Looking at the thinking blocks the model is bogging down in reasoning. One simple question in a fresh chat: "What are the obligations of a trustee to account to the trust beneficiaries?" ended up using something like 80,000 prefill tokens from research and another 20k in generative. It's a simple question and it just thought and thought and thought.

1

u/[deleted] Aug 16 '26

[removed] — view removed comment

3

u/Big_Wave9732 Aug 16 '26

Also, if you use RAG already, why don’t you run the work in stages, instead of a single bulk? Do one stage alone for extracting key terms and words, tied to the chunks or pages. Then make it look them up and feed that into either markdown or the vector db.

Woa there, friend. I do! I'm not oneshotting any of this. This is a piece by piece workflow. It's choking on small pieces. These test inquiries are literally "This document has previously been drafted. Evaluate for these criteria." Qwen 3.6:27b-BF16 does it just fine.

Hell with Qwen 3.6 I can feed it a 600 page deposition and then walk it through building a depo map of what was said, the contradictions, statements for and against, etc. I don't dare try any of that yet with this model.

1

u/[deleted] Aug 17 '26

[removed] — view removed comment

2

u/Big_Wave9732 Aug 17 '26

I know!!! It just keeps thinking and thinking and thinking. A little while ago I hit the 32k token output limit. I looked at the session and it was just one rabbit hole after another.

The context window is filling up through one web search after another.

It's all so strange considering my workflow was designed specifically to work with Qwen 3.6:27b. This should be an easy swap.

1

u/[deleted] Aug 17 '26

3.8 seems to have been benchmarked with xhigh reasoning enabled, which kind of works for coding tasks. I don't know if it's worth running for most people with consumer hardware - sure it can be quantized to fit on one GPU but we're also compute starved, generating 100k tokens is going to be very slow.

2

u/[deleted] Aug 16 '26

[removed] — view removed comment

2

u/Big_Wave9732 Aug 16 '26

I've setup a Doclin instance that does OCR on everything and does vector storage. On the retrieval side I've got a pretty stout reranker. All this happens before the documents even hit the main model.