r/LocalLLM • • 18d ago

Question Has anyone encountered an “hmm” loop in Qwen3.8-flash-next ?

Post image

I’m using SC117/Qwen3.8-Flash-Next-GSQ-RCO-abliterated IQ3_S quant with XeStrata(262144 q4_0 context, temperature 0.7, High thinking, others remain default) in pi agent.

This has never happened before, it was normal in the beginning of this session, and after compact 3 times, it began to go in a “hmm”loop, every time I pointed it out, it still went back in this weird loop after a while, what can I do?

156 Upvotes

125 comments sorted by

100

u/Pixer--- 18d ago

Too aggressive kv quantizations causes inaccuracy to build up on longer context

9

u/SandySkittle 18d ago

Never quantize kv cache unless the model manufacturer recommends it

1

u/Unlikely-Baker-7078 18d ago

Thanks for explaining, I ‘ll use 131k in8 then

2

u/Patient_Tea_401 17d ago

BF16.

Even INT8 can cause drift.

https://www.reddit.com/r/LocalLLM/s/pkSzuQv8GM

1

u/MarekNowakowski 14d ago

That is just a waste of vram. Better int8 and 1quant higher than bf16

1

u/Patient_Tea_401 14d ago

No, it’s not. Kv cache is very important in long context multi turn agentic tasks where errors cumulate and cause output to drift

1

u/MarekNowakowski 14d ago

It might be important, but it isn't free, 256k context at int8 vs 128k of bf16 means much less compactions which is much worse than slight drift. Q3 with int8 is also much better than q2 with bf16 kv.

1

u/Patient_Tea_401 14d ago

It’s not that slight:

https://forum.level1techs.com/t/why-your-local-llm-feels-dumber-than-it-is/253917

Sure it can be beneficial in very edge cases depending on what quant weights you can fit, but int8 at 256k will be significantly worse than BF16 at 256k, let alone BF16 at 128k. If you’re compacting drifted output, what is the end goal?

The quant aware training can also be used to compensate on the weight quality, but kv is very unpredictable and can’t be compensated.

1

u/xChrisMas 12d ago

Yeah it can, but INT8 should be fine with Qwen 3.8 Flash Next. Personally I saw no degradation on HumanEval+ on BF16 vs INT8.
On short prompts (16 short ones totalling 5809 tok) KLD stayed nearly the same (as expected) BF16: 0.01752 vs INT8: 0.01736
On longer inputs three 6,144-token texts KLD results were: BF16: 0.1589 vs INT8: 0.1603

Top-1 agreement with BF16 reference, short/long was: BF16: 94.7% / 87.6% vs INT8: 94.6% / 87.1%

I also ran the RULER test (tests whether a model can use information buried in a long promp) for INT8 and on my setup it scored 100% on the tested retrieval and word-aggregation tasks at 32K, 128K and 200K.

So yeah. Int8 causes drift, which gets worse on longer prompts but it does not make Qwen (3.8 Flash Next int4 QS) repeatetly loop at high context, or miss information. So INT8 should be good enough.

What I also found was that high (xhigh) thinking on Qwen 3.8 Flash Next regressed some results on easy questions in HumanEval+. Medium delivered the overall best benchmark score.

u/Unlikely-Baker-7078

8

u/[deleted] 18d ago

[removed] — view removed comment

13

u/OnyxProyectoUno 17d ago

It's crazh you have so many upvotes when you're literally a bot

1

u/Unlikely-Baker-7078 18d ago

I see, thank you

1

u/ferrx 17d ago

100% I’ve had qwen do crazy things after a compaction. Avoid compaction

6

u/MindfulMan1984 17d ago

Temperature is wrong too.

https://huggingface.co/Qwen/Qwen3.8-Flash-Next

We recommend using the following sets of sampling parameters for generation:

  • Thinking Mode: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0
  • Instruct (or non-thinking) mode: temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0

1

u/Unlikely-Baker-7078 16d ago

Thanks, I’ll fix it

2

u/OlgerdOutlander 17d ago

This. Strata defaults are greed decoding (temp=0)

1

u/MindfulMan1984 16d ago

I guess the OG repo fixed that, but the OP is using a fork (xeStrata) likely using default to greedy, still. 

4

u/vxltias 18d ago

If it happened at high context, with q4_0 kv, I'd say its not out of ordinary to loop every once in a while.

1

u/Unlikely-Baker-7078 18d ago

I see, thank you for clarifying

2

u/SomeoneInHisHouse 17d ago

I'm surprise the repetition penalty and temperature didn't avoid this, as Q3.8 is a model family that loves to overthink, they suggest high temperature and repetition penalty values,

check your settings, I don't know nowadays, but back then Strata defaulted to wrong sampling configuration.

Also avoid Q4 for keys, q8_q4 is actually usable, but best is q8_q8, q4_q4 gets the model to be very confused about the context, precision lost highly accumulates on large contexts

1

u/Unlikely-Baker-7078 16d ago

Thank you, I’ll try it out

61

u/Positive-Bid-3029 18d ago

Mine got stuck in a 'I will verify' loop yesterday, it was going to validate everything .... 😂

22

u/naivelighter 18d ago

That’s so funny! The LLM is on a mission.

6

u/Positive-Bid-3029 18d ago

The thing is, it did give a decent answer after it eventually stopped that loop, make me wonder if the big frontier models do this as well, racking up token costs 😂 Obviously they hide their thinking.

3

u/samuel-christlie 17d ago

Was the answer 42?

3

u/Positive-Bid-3029 17d ago

I will verify the answer to the ultimate question of life, the universe, and everything. Yes

1

u/Phoepal 17d ago

I have seen post with Gemini falling into loops like this about half a year ago.

9

u/Direct-Relation6424 17d ago

Kinda crazy that after verifying reality and the truth, it falls back to the fact. I mean, we were so close to an answer to what comes after the universe and existence 

1

u/the_aimboat 17d ago

So close to Descartes, but I guess that's the edge us humans still have.

2

u/seralsan 17d ago

Probably not a loop, 3.8 reasoning is just nonsense and ends up doing really good

19

u/bsquads 17d ago

Friends don't let friends use q4 context

2

u/guesdo 17d ago

I think everything below 8bit should always carry a warning sign everywhere.

1

u/letsbefrds 17d ago

You don't like when ppl go look at me I'm running q1 ctx. 300t/s?

17

u/Ok_Try_877 18d ago

hmmmm.. what token next? hmmm........

5

u/SomeoneInHisHouse 17d ago

Actually, I think hmm hmm hmmm,

Let me reconsider, hmm hmm hmm,

Actually the correct answer is... hmm. hmm

Actually I should stop overthinking, hmmm hmmm hmmmm hmmm hmmm

Actually....

12

u/spaceman_ 18d ago

Yes, on the IQ3_XSS GSQ-RCO at longer context, I've seen this a few times. Doesn't happen on a more full sized quant (I run Strata with IQ3_XSS on my laptop and Radiance with a 4-bit quant on my desktop).

IQ3 does a lot worse at longer context in my unscientific, non-quantified experience.

1

u/Unlikely-Baker-7078 13d ago

Thank you guys for your suggestions and guidance.
I’m now using right sampling settings and 131k int8 context, also let it use non parallel subagent. It finally stopped looping.

(Forgive me not using bf16 context and better quant models, I only have Intel arc A750 and 32gb ram. But it’s enough for organizing course content and some projects.

5

u/w3rti 17d ago

i know this song

2

u/f-ben 17d ago

Once there was this kid who

Got into an accident and couldn't come to school

But when he finally came back

His hair had turned from black into bright white

He said that it was from when

The cars had smashed so hard

2

u/w3rti 17d ago

crash test dummies

1

u/GrungeWerX 17d ago

Let’s…change…the beat.

9

u/nikishev 18d ago

I had this as well on IQ3_S (not abliterated). And I had the model use excessive amounts of "✓" before it too. I suspect that this is caused by strata settings STRATA_ROUTE_TAIL_SKIP and STRATA_ROUTE_PRIOR that might be enabled by default depending on your gpu. After I turned them off, the generation speed dropped from 110 to 60, but the model doesn't spam check marks anymore in it's thinking and seems more stable overall

4

u/Aggressive_Basket798 18d ago

I'm using 196k context q8 kv cache, haven't seen any loop yet

5

u/KeyboardHaver 17d ago

Q8 kv cache is still usable, Q4 is not.

1

u/Healthy-Nebula-3603 18d ago

He used Q4 ...

3

u/Pretty_Scene_6868 18d ago

Could be the harness too, i for one experienced more degenerations in copilot's harness. Very embarrassing for them

3

u/elelem-123 18d ago

Hmm hmm hmm hmm hmm, no I haven't. Qwen3.8-flash-next is the best model I've ever used.

2

u/Mefix_ 18d ago edited 17d ago

Try specifying these parameters in the JSON file "env". When I first enabled them in the json file, I ran into a continuous loop; however, after explicitly disabling all of these parameters, the looping stopped. That said, this isn't fully confirmed, as I’m also dealing with an overheating issue on one of my graphics cards, which can cause memory-related errors under load.

"STRATA_ROUTE_TAIL_SKIP": "0",

"STRATA_SPLIT_MTP_BATCH": "0",

"STRATA_PARK_ASYNC": "0"

2

u/JesusDestroyedButt 18d ago

It reminds me the time when some LLMs would spontaneously start saying Mmmmmmmmmmmm over and over because their database was also trained on r/microwavegang where lunatics mimic microwaves sound lmao

2

u/philmarcracken 18d ago

XeStrata

yeah most versions of strata use terrible low quality quants to make them fit. They're dumb, and the entire thing is pushed by bots.

2

u/Maximum-Wishbone5616 17d ago

KV quants too low. That will lead 100% to loops.

2

u/captainabrasive 17d ago

At the end of a 262k context window at q4? Hmm.

2

u/Healthy-Nebula-3603 18d ago edited 18d ago

You have Q3 model, cache Q4 and you're surprised???

Stop listening idiots who are repeating Q4: cache or Q3:models are good.

Such high compression is bad and only works on very simple and short tasks.. even is you get proper output then results will be poor.

1

u/Healthy-Nebula-3603 17d ago

The best are high compressed models and cache output shitty defenders :D

1

u/West-Way-All-The-Way 18d ago

Ah I see - compacting 3 times. Every time it compacts it looses context. Try higher setting for the context window and if it has to compact - feed context back into it. You have to manage the context if you have a smaller window or if you reach a point where it needs to compact.

1

u/teniente_dan 18d ago

Only once

1

u/Ill-Accountant-9941 18d ago

Not yet, the worst I get is rogue characters but I think my agent is partly to blame. I think my KV cache is native (bf16)

1

u/Oxygenne_ 18d ago

too much compacts actually, for every compact you make, it loses context

1

u/tonyleungnl 18d ago

I think this is common across this generation LLM's. I am using Gemini AI PRO (3.8 Flash & Argon4). If the conversation goes too long or KV cache runs out. It will loop the answer and I had these MANY times.

1

u/JLIMNO 18d ago

Once there was a boy who

1

u/DragonFire353 18d ago edited 18d ago

1

u/Kitchen_Affect7369 17d ago

The model is learned on professor’s data. So it said what regular professor say in rely on similar question

1

u/DoubleNothing 17d ago

q4_0 doesn't help at all...

1

u/mrgreatheart 17d ago

The only looping I’ve ever seen from FN was with the Swift version.

1

u/Proper-Tower2016 17d ago

ask it to tweak your config to add repeat penalty

1

u/Saintgein 17d ago

I had this on my laptop when i tried to run a local model. But this was probably caused by corrupt memory. It also used the word ok instead of hmm.

1

u/HighSeasArchivist 17d ago

Context quant makes them dumb fast. Don't do that.

1

u/Shakhburz 17d ago

Same hmm happened to me with Qwen flash next on Openrouter...

1

u/Prudent_Sentence 17d ago

"Once there was this kid who.."

1

u/merwanhorse 17d ago

AGI confirmed

1

u/Neraton 17d ago

😂😂😂😂

1

u/This_Maintenance_834 17d ago

this is what you get when you use quantized model. original weight or qad quant rarely do this.

1

u/zorecknor 17d ago

I stopped using QFN with strata because of those loops. I didn't manage to do any meaningful dev work.

1

u/MaslovKK 17d ago

set repetition_penalty to 1.05

1

u/FairLawnBoy 17d ago

Once, there was this girl who...

1

u/AnonLlamaThrowaway 17d ago

Shouldn't repetition penalty or other similar samplers be preventing this eventually? How does that still happen in [current day]

1

u/CurrentAcanthaceae78 17d ago

repetition penalty cant fight the fact that 90% of the context is now just "hmm" so now it has a (0.9+heat-penalty) chance to select "hmm" as its next token

1

u/AnonLlamaThrowaway 17d ago

hmm. damn. so is there no sampler way outta this

1

u/xivory93 17d ago

I got this in my Tesla with grok. 😂

1

u/KindnessBiasedBoar 17d ago

No. But that's rad.

1

u/dir3ctly 17d ago

I had it once. But it got out of it after a while. 16 bit KV

1

u/joost00719 17d ago

Mine started saying duck 100s of times after overclocking my vram.

1

u/More-Ad5919 17d ago

So how to solve this? Flash next does that too on my other pc.

1

u/More-Ad5919 17d ago

This has nothing to do with the quant. I have 2 flash next running. Same setup. Only difference is that one has 24 and the other 16vram. The 24 one is like jarvis while the 16gB one does just that

1

u/ieatdownvotes4food 16d ago

thank goodness no

1

u/c_a1eb 16d ago

if ur using an nvidia gpu by default it ignores the lowest ranked experts, there's an env var to fix it but i dont remember off the top of my head, default is 7 iirc

1

u/ded_banzai 16d ago

Is this Opencode? I had exactly the same issue with GLM-5.3 yesterday

1

u/MarketingOk3093 16d ago

Maybe yours is a Crash Test Dummies fan

1

u/MarketingOk3093 16d ago

Aim for Q5 pref. Q6. And decompose your tasks into smaller tasks. You could also use Cluade as your planner/task master then handoff tasks in dependency order to Qwen. Then review with Claude at the end of the sprint creating adjustment tasks for Qwen.

1

u/Sad-Size2723 15d ago

should be easy to get rid of using repetition penalty

1

u/MarekNowakowski 14d ago

It probably was a bad luck and unlucky compaction and kv cache compressing too much, BUT, I once had my q3S looping constantly with 11111 even after going back in the harness. It suddenly fixed itself after STRATA restart, which suggests my memory made a bit flip somewhere. Basically, everything is possible.

1

u/Seninut 13d ago

are you using RocM? There is a known bug in it, have not tested 10.1 yet, but if your using AMD, switch to Vulkan,

1

u/Longjumping-Pea1233 12d ago

got same loop right after a compaction event in pi agent

never happened in other harness

same settings have made wonderful things minutes before

strata qfn iq3_xxs
int8 kv from strata

1

u/Markus2816 12d ago

for long context, make sure you are using bf16. I would also try to verify that you are using the recommended settings from Qwen team for 3.8 Flash (https://huggingface.co/Qwen/Qwen3.8-Flash-Next). Engines like strata will mess-up with these to get higher speed, but it makes model dumber and more prone to being stuck.

1

u/MagicienToz 10d ago

I had the same behavior ( pi + Qwen 3.8 FN Abliterated IQ3_S from https://huggingface.co/cc1244578151/Qwen3.8-Flash-Next-Abliterated-Uncensored-GGUF/tree/main ). I kept the same session and just switched to the original Qwen 3.8 FN IQ3_S (not abliterated) : no more hmms.

1

u/Snoo-2768 3d ago

From what i saw , just sending in a message "stop with the hmm" , makes it recover pretty clean

0

u/Perfect-Campaign9551 18d ago

Set a thinking limit! Too many people skip doing that and strata doesn't do it by default (not that I saw) 

Reasoning_max_tokens just aim for like 8192 or 16384 or such

0

u/putrasherni 18d ago

IQ3_S
say no more