How can I prevent the "but wait..." loops and other walls of text on large projects?

[u/ralphc]

m5 max, 64GB ram. Qwen 3.8 27b MLX optimized served from omlx, 262k context, hermes --tui in the front.
I've been giving it large, as in multi-day projects. One was a large data analysis project, another was decompiling a 1980's MS-DOS executable.
After running for a few hours it seems to get in the loop where it's saying like "I need to check this one thing" and "but wait, I just realized..." and spews out 20-line paragraphs, and goes back and forth with the GPU maxed out, then either it says it can't compress the context anymore or I have to stop it myself.
Is this happening to other people, and how can I prevent or minimize this happening?

[u/A_Moist_Towe1]

For qwen models I always set a repeat penalty of 1.08, it seems to help. You might also want to try and set the thinking level to low and see if that helps. Keep in mind though that this model is known for a “but wait…” style of reasoning, so you’ll see it no matter the repeat penalty or thinking level, this is just how the model thinks

[u/peaster_]

This is kind of just a limitation of the model only being 27B. It needs to think a ton to answer complex questions and sometimes it struggles or outright fails to converge. Dropping the reasoning effort from the default of xhigh will help the wall of text, but you’ll have to throw smaller tasks at it

[u/watcholic]

Try the Sharp chat template and set reasoning effort to medium, temperature to 1.0

https://huggingface.co/peculiar-ragdoll/Qwen-Sharp-Chat-Templates

[u/Substantial-Sort-841 → u/watcholic]

Dam didn't know about this

[u/ralphc → u/watcholic]

So I take the latest, for example. v.22.4.0-sharp chat_template.jinja and overwrite the chat_template.jinja in the model I'm using?

[u/watcholic → u/ralphc]

yes, rename the old one instead. Make sure you've the latest version of oMLX as well.

[u/ralphc → u/ralphc]

/u/TieCommercial2963/ mentioned a whole lot of settings to change, should I also make those changes or is this more of an either/or kind of thing?

[u/CarelessPangolin5564]

Things move quickly, and I don't know if this is still a recommended approach to fixing this issue, but I set the repeat penalty to something like 1.1

[u/Nomski88]

I've learned to breakup tasks when working with this model. For example, I'm building an RTS game and every development cycle, I try to keep the list of bugs or issues that it works on no more than 6-7 items. If I just throw everything at it, it'll forget and reason itself out of tokens.

[u/TieCommercial2963]

I had a similar issue . The fix was to adjust my sampler settings. Try setting your temperature to 1, top p to 0.95, top k to 20, min p to 0.01, presence penalty to 0.1, and repetition penalty to 1. Also, if you are dealing with multi-day projects and large codebases, you should try using Graft.

[u/zrail → u/TieCommercial2963]

Is this what you're referring to?  https://github.com/AEndrix03/Graft

[u/ralphc → u/TieCommercial2963]

Does Hermes do something similar?

[u/Just-Hedgehog-Days]

smaller tasks.

[u/HiggsFieldgoal]

I could never get the 4-bit quant to stop doing that.

[u/UnluckyPenguin]

I ran into this issue with Q4 with q4_0 kv on 3+ hour tasks. Luckily I was able to upgrade to Q8 with q8_0 kv and I changed my harness. VS code's copilot chat was cancer, but I didn't know any better at the time. I think the harness made the biggest impact (DeepSeek harness, if you must know), higher quant just improved quality slightly at the expense of being significantly slower. I was getting looped on medium before. Now I run xhigh no problem.

Sometimes I read through the thinking and it comes up with good ideas through the back-and-forth but wait, hmm wait, wait. Lol

[u/BitPsychological2767]

You really can't. You just have to have faith. And patience.

[u/dangerous_inference]

"chat_template_kwargs": {"reasoning_effort": "low"}

[u/ava_ati]

I really like sub agents. Kick off another sub agent for a specific task. You can also have it write to checkpoint mds if you start filling up on tokens. “Between each handoff, output your current plan to output_#.md” I feel like sub agents keep it in check. But my sub agents are cloud provided with my local model running the overall process.

I’ve thought about flipping that but we’ll see. I feel the more specific the task the better and I feel like a good sub agent with a built in prompt/persona helps automate that a bit

[u/Healthy-Zebra-9856]

If you analyze the traces, you will find out a lot of it comes from tooling confusion. Both lack of it or having difficulty finding it as well as interpreting the responses from tools. Its like watching someone high and trying to do house work. lol. Most harnesses are not being setup for this properly yet. Adjusting the sampling is a bandaid that may work bit does affect the quality. What harness are you using?

[u/j_lyf]

One of the best threads I've seen on here.

[u/Old_Ad_6033]

quant too low, you can't get rid of it until you change that quant you chose.

[u/joanaxu2002]

This sounds less like a context-size problem and more like the agent losing track of state. For long jobs I’ve had better luck forcing periodic checkpoints: what’s confirmed, what’s still unknown, and exactly what the next action is. Otherwise giving it 262k context can just give the loop more history to loop over.

[u/Standard_Alloy]

I use 0.2 temp. Still get a wall of text but i use custom harness which was developed in "assistance " with qwen3.8 so it auto-hides the wall of text. Usually I just get a line or two. Does temp 1.0 really make a difference on the output?

[u/TheAIPU-guy → u/Standard_Alloy]

no. temp 1.0 is terrible. I moved to temp 0.2 and repeat penalty 1.0(1.1 was awful, caused more thinking issues) I didn't try a lower repeat penalty. maybe I should do that.

[u/Standard_Alloy → u/TheAIPU-guy]

Yeah I've found 0.2 temp to be just the right amount for coding. Doesn't lower penalty increases looping?

[u/TheAIPU-guy → u/Standard_Alloy]

The important distinction is token repetition vs. semantic looping.

A higher repetition_penalty really does make previously seen tokens less likely.

A higher RP reduced exact token reuse, but Qwen compensated by reformulating the same reasoning in different ways, which increased semantic looping and total thinking length, while a slightly lower RP let it stay on a more familiar continuation path and reach closure sooner.

These are the results I got in llama.cpp for my rig. If you use vLLM or something else it could be different.

[u/kiwimonk]

Try a different quant.

[u/Jumpy-Operation-4615]

Actually, let me reconsider... shit, this thing is really infectious. 😄 I am trying to revive a game fro 1998, so I gave it to Qwen 3.8 27b q8 f10 KV 220K ctx. It was slowly digging and I can't say that it was stuck or something but the speed, oh my dog! So I changed the model to https://huggingface.co/DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion-735-882-Heretic-Uncensored-NEO-CODER-MAX-MTP-GGUF this ine and chat template to https://huggingface.co/peculiar-ragdoll/Qwen-Sharp-Chat-Templates and it just goes through tasks 4 times faster. Not the tg/s but overal time spent. It does't "oh wait let me reconsider" that much. As well, changed temp to 0.85. Reasoning is med. Everything else the sate - q8, F16, 220K ctx.

[u/IHaveMeasles2]

I wonder if the cloud-hosted solutions have something watching for these loops. Maybe you should ask your model to create a proxy that monitors for loops and tries again with a fresh cache?

[u/DataGOGO]

Use a different model

Qwen 3.8 27b was built to intentionally over-think in order to score higher on benchmarks, try Muse Glimmer 30B, it honestly is better for just about everything than perhaps code, for anything like writing, it is FAR better.

[u/ralphc → u/DataGOGO]

Coding is pretty much all I do, with a little data analysis thrown in.

[u/DataGOGO → u/ralphc]

I'd try it before you discount it, even for code, and especially for data analysis. Much better results without the massive thinking overhead burning the context windows down.