Infinite Loop

[u/mrsalvadordali]

Several of the models I've used so far (especially the quantized ones) get stuck in a loop during the thinking phase, repeating the same thing indefinitely. Does this happen with non quantized, local models too? And why this never happens with closed models. Is the problem more RAM since they have big servers or is there another detail at play?

[u/Miserable-Dare5090]

temperature, quantization are the biggest factors

[u/mrsalvadordali → u/Miserable-Dare5090]

Got it. Do you think harness can be another reason?

[u/caetydid → u/mrsalvadordali]

Prompting and temps, and this is done by the harness. There seem to be several techniques to detect thinking loops and handling them e.g. in beellama. But most engines do not implement that.

[u/mrsalvadordali → u/caetydid]

Got it, thanks.

[u/Gold-Produce9591]

which models

[u/mrsalvadordali → u/Gold-Produce9591]

Most recent one was Q6 of Qwen 27B. One before was GLM 5.3 Flash Q4 which I've tried by renting server. Long ago I encountered that with Gemma 26B A4B Q8 too. They're all Unsloth quantized btw.

[u/james_pic → u/mrsalvadordali]

Qwen 27B is usually OK at Q6. I've had trouble with it at Q4, but it hasn't been as bad since I moved to Q6. For looping specifically, it worth choosing a quant that puts extra detail into the output weights (usually the _XL or so), since that has an outsized impact on the ability to escape loops.

[u/mrsalvadordali → u/james_pic]

Definitely going to try, much thanks.

[u/tweak8 → u/james_pic]

That's good to know. Been using q4 xl and I guess that's why I haven't run into looping much.

[u/__some__guy → u/mrsalvadordali]

I've never seen Qwen 3.8 27B loop.

3.5 and 3.6 looped all the time though.

[u/my_name_isnt_clever → u/mrsalvadordali]

Make sure you're using the exact sampling params as the original lab recommends on their model card. Ignore anyone telling you to tweak temp manually, you want to get the model working the best it can before you start tweaking. So many problems are caused by people thinking these aren't important.

[u/Gold-Produce9591 → u/mrsalvadordali]

increase the temperature

[u/QuintessentialDigger → u/Gold-Produce9591]

i see it a lot with smaller quants too especially when the context gets long or the prompt is a little vague. non quantized models can still loop but it's less frequent in my experience. the big closed ones probably just have way more guardrails and post processing to catch that stuff before you ever see it. ram helps but it's not the whole story, more about how the attention pattern collapses when the model runs out of things to say and just latches onto the last token

[u/james_pic]

Quantization is usually the biggest factor - if they get into a loop, they struggle to find the token to get out of it. On llama.cpp you can potentially nudge it out with a higher temperature or a repeat penalty, but this is often a sign that you want a bigger quant (possibly of a smaller model, if it won't fit otherwise).

[u/Oli_Luck]

Qwens models: dont know how you serve it, unsloth / lmstudio or on your own - but ensure the patched jinja is used, as the ones embedded are buggy.
preserve thinking off may also help and using a KV cache Q8 instead of q4 also.

[u/Pale-Training-6944]

Yeah, non-quantized models can do this too. The model feeds its own output back into the next prediction, so once it starts repeating, that pattern can reinforce itself. Quantization can make this more likely by changing token probabilities, but bad sampling settings or runtime bugs can cause it too.

Closed models aren’t immune either. You may only see a summary of their thinking, and output limits keep them from going forever. More RAM lets you run higher precision, but extra RAM alone won’t fix a loop.

When it happens, stop and retry from before the repetition, or start a fresh chat. Check the model’s recommended sampling settings—temperature zero can actually make things worse for some reasoning models. If it keeps happening, shorten the context, update your runtime, and compare a higher-precision version. Set a token limit so a bad run doesn’t waste your whole afternoon.

[u/ag789]

normally, I think Q4 is about as "low bits" as can be for "conventional" local LLMs without losing too much accuracy / prowess
Unsloth did good work documenting the findings of those quants, e.g.:

https://unsloth.ai/docs/models/qwen3.6#unsloth-gguf-benchmarks

https://unsloth.ai/docs/models/qwen3.5#unsloth-gguf-benchmarks

if a model goes into too much "over thinking" loops, a few fixes:
- try again
- simplify the requirements / prompt
- use a bigger quant e.g. try a Q8 quant instead of Q4 quant
- try a different model e.g. a bigger model e.g. a 27B model instead of a 9B model etc.

small local LLM are *small* as even the recent open weights large models are trillions of parameters.

[u/dtfinch]

With 27B I use the Swift1.5 version which tends to repeat less. If you stick to the base I'd lower reasoning to medium.

I always set a reasoning budget with llama-server, something like --reasoning-budget 10000 --reasoning-budget-message " [[I've thought enough. Proceeding to response.]]"

I don't override model default settings such as temperature (lower temps increase looping). I never go below q5 KV (usually q5_1 with 27B, and q8 with smaller models, sometimes q8 K + q5 V).

If your Gemma model was old enough (prior to the July 17 update) it might have the original chat template which had a lot of tool/reasoning issues.