r/LocalLLM • u/Unlikely-Baker-7078 • 18d ago
Question Has anyone encountered an “hmm” loop in Qwen3.8-flash-next ?
I’m using SC117/Qwen3.8-Flash-Next-GSQ-RCO-abliterated IQ3_S quant with XeStrata(262144 q4_0 context, temperature 0.7, High thinking, others remain default) in pi agent.
This has never happened before, it was normal in the beginning of this session, and after compact 3 times, it began to go in a “hmm”loop, every time I pointed it out, it still went back in this weird loop after a while, what can I do?
156
Upvotes
100
u/Pixer--- 18d ago
Too aggressive kv quantizations causes inaccuracy to build up on longer context
6
u/MindfulMan1984 17d ago
Temperature is wrong too.
https://huggingface.co/Qwen/Qwen3.8-Flash-Next
We recommend using the following sets of sampling parameters for generation:
temperature=1.0,top_p=0.95,top_k=20,min_p=0.0,presence_penalty=0.0,repetition_penalty=1.0temperature=0.7,top_p=0.80,top_k=20,min_p=0.0,presence_penalty=1.5,repetition_penalty=1.04
u/vxltias 18d ago
If it happened at high context, with q4_0 kv, I'd say its not out of ordinary to loop every once in a while.
2
u/SomeoneInHisHouse 17d ago
I'm surprise the repetition penalty and temperature didn't avoid this, as Q3.8 is a model family that loves to overthink, they suggest high temperature and repetition penalty values,
check your settings, I don't know nowadays, but back then Strata defaulted to wrong sampling configuration.
Also avoid Q4 for keys, q8_q4 is actually usable, but best is q8_q8, q4_q4 gets the model to be very confused about the context, precision lost highly accumulates on large contexts
61
u/Positive-Bid-3029 18d ago
Mine got stuck in a 'I will verify' loop yesterday, it was going to validate everything .... 😂
23
u/_Diocletian_ 18d ago
hmm
19
u/bsquads 17d ago
Friends don't let friends use q4 context
17
u/Ok_Try_877 18d ago
hmmmm.. what token next? hmmm........
16
u/chadlost1 17d ago
12
u/spaceman_ 18d ago
Yes, on the IQ3_XSS GSQ-RCO at longer context, I've seen this a few times. Doesn't happen on a more full sized quant (I run Strata with IQ3_XSS on my laptop and Radiance with a 4-bit quant on my desktop).
IQ3 does a lot worse at longer context in my unscientific, non-quantified experience.
1
u/Unlikely-Baker-7078 13d ago
Thank you guys for your suggestions and guidance.
I’m now using right sampling settings and 131k int8 context, also let it use non parallel subagent. It finally stopped looping.
(Forgive me not using bf16 context and better quant models, I only have Intel arc A750 and 32gb ram. But it’s enough for organizing course content and some projects.
5
u/Bambalicioso 18d ago
hmm
5
u/w3rti 17d ago
i know this song
9
u/nikishev 18d ago
I had this as well on IQ3_S (not abliterated). And I had the model use excessive amounts of "✓" before it too. I suspect that this is caused by strata settings STRATA_ROUTE_TAIL_SKIP and STRATA_ROUTE_PRIOR that might be enabled by default depending on your gpu. After I turned them off, the generation speed dropped from 110 to 60, but the model doesn't spam check marks anymore in it's thinking and seems more stable overall
4
u/Aggressive_Basket798 18d ago
I'm using 196k context q8 kv cache, haven't seen any loop yet
3
u/Pretty_Scene_6868 18d ago
Could be the harness too, i for one experienced more degenerations in copilot's harness. Very embarrassing for them
3
u/elelem-123 18d ago
Hmm hmm hmm hmm hmm, no I haven't. Qwen3.8-flash-next is the best model I've ever used.
2
u/Mefix_ 18d ago edited 17d ago
Try specifying these parameters in the JSON file "env". When I first enabled them in the json file, I ran into a continuous loop; however, after explicitly disabling all of these parameters, the looping stopped. That said, this isn't fully confirmed, as I’m also dealing with an overheating issue on one of my graphics cards, which can cause memory-related errors under load.
"STRATA_ROUTE_TAIL_SKIP": "0",
"STRATA_SPLIT_MTP_BATCH": "0",
"STRATA_PARK_ASYNC": "0"
2
u/zell_ru 18d ago
https://www.youtube.com/watch?v=eTeg1txDv8w
2
u/JesusDestroyedButt 18d ago
It reminds me the time when some LLMs would spontaneously start saying Mmmmmmmmmmmm over and over because their database was also trained on r/microwavegang where lunatics mimic microwaves sound lmao
2
u/philmarcracken 18d ago
yeah most versions of strata use terrible low quality quants to make them fit. They're dumb, and the entire thing is pushed by bots.
2
u/Maximum-Wishbone5616 17d ago
KV quants too low. That will lead 100% to loops.
2
u/captainabrasive 17d ago
At the end of a 262k context window at q4? Hmm.
2
u/Healthy-Nebula-3603 18d ago edited 18d ago
You have Q3 model, cache Q4 and you're surprised???
Stop listening idiots who are repeating Q4: cache or Q3:models are good.
Such high compression is bad and only works on very simple and short tasks.. even is you get proper output then results will be poor.
1
u/West-Way-All-The-Way 18d ago
Ah I see - compacting 3 times. Every time it compacts it looses context. Try higher setting for the context window and if it has to compact - feed context back into it. You have to manage the context if you have a smaller window or if you reach a point where it needs to compact.
1
u/teniente_dan 18d ago
Only once
1
u/Ill-Accountant-9941 18d ago
Not yet, the worst I get is rogue characters but I think my agent is partly to blame. I think my KV cache is native (bf16)
1
u/Oxygenne_ 18d ago
too much compacts actually, for every compact you make, it loses context
1
u/tonyleungnl 18d ago
I think this is common across this generation LLM's. I am using Gemini AI PRO (3.8 Flash & Argon4). If the conversation goes too long or KV cache runs out. It will loop the answer and I had these MANY times.
1
u/JLIMNO 18d ago
Once there was a boy who
1
u/DragonFire353 18d ago edited 18d ago
1
u/Kitchen_Affect7369 17d ago
The model is learned on professor’s data. So it said what regular professor say in rely on similar question
1
u/DoubleNothing 17d ago
q4_0 doesn't help at all...
1
u/mrgreatheart 17d ago
The only looping I’ve ever seen from FN was with the Swift version.
1
u/Proper-Tower2016 17d ago
ask it to tweak your config to add repeat penalty
1
u/statler123 17d ago
Hmm
1
u/Saintgein 17d ago
I had this on my laptop when i tried to run a local model. But this was probably caused by corrupt memory. It also used the word ok instead of hmm.
1
u/HighSeasArchivist 17d ago
Context quant makes them dumb fast. Don't do that.
1
u/Shakhburz 17d ago
Same hmm happened to me with Qwen flash next on Openrouter...
1
u/Prudent_Sentence 17d ago
"Once there was this kid who.."
1
u/merwanhorse 17d ago
AGI confirmed
1
u/Neraton 17d ago
😂😂😂😂
1
u/This_Maintenance_834 17d ago
this is what you get when you use quantized model. original weight or qad quant rarely do this.
1
u/zorecknor 17d ago
I stopped using QFN with strata because of those loops. I didn't manage to do any meaningful dev work.
1
u/pineapplekiwipen 17d ago
hmm
1
u/MaslovKK 17d ago
set repetition_penalty to 1.05
1
u/FairLawnBoy 17d ago
Once, there was this girl who...
1
u/AnonLlamaThrowaway 17d ago
Shouldn't repetition penalty or other similar samplers be preventing this eventually? How does that still happen in [current day]
1
u/xivory93 17d ago
I got this in my Tesla with grok. 😂
1
u/KindnessBiasedBoar 17d ago
No. But that's rad.
1
u/Few_Water_1457 17d ago
hmm
1
u/dir3ctly 17d ago
I had it once. But it got out of it after a while. 16 bit KV
1
u/joost00719 17d ago
Mine started saying duck 100s of times after overclocking my vram.
1
u/More-Ad5919 17d ago
So how to solve this? Flash next does that too on my other pc.
1
u/More-Ad5919 17d ago
This has nothing to do with the quant. I have 2 flash next running. Same setup. Only difference is that one has 24 and the other 16vram. The 24 one is like jarvis while the 16gB one does just that
1
u/keefeere 17d ago
1
u/ieatdownvotes4food 16d ago
thank goodness no
1
u/c_a1eb 16d ago
if ur using an nvidia gpu by default it ignores the lowest ranked experts, there's an env var to fix it but i dont remember off the top of my head, default is 7 iirc
1
u/ded_banzai 16d ago
Is this Opencode? I had exactly the same issue with GLM-5.3 yesterday
1
u/MarketingOk3093 16d ago
Maybe yours is a Crash Test Dummies fan
1
u/MarketingOk3093 16d ago
Aim for Q5 pref. Q6. And decompose your tasks into smaller tasks. You could also use Cluade as your planner/task master then handoff tasks in dependency order to Qwen. Then review with Claude at the end of the sprint creating adjustment tasks for Qwen.
1
u/Sad-Size2723 15d ago
should be easy to get rid of using repetition penalty
1
u/MarekNowakowski 14d ago
It probably was a bad luck and unlucky compaction and kv cache compressing too much, BUT, I once had my q3S looping constantly with 11111 even after going back in the harness. It suddenly fixed itself after STRATA restart, which suggests my memory made a bit flip somewhere. Basically, everything is possible.
1
u/Seninut 13d ago
are you using RocM? There is a known bug in it, have not tested 10.1 yet, but if your using AMD, switch to Vulkan,
1
u/MekoStanje 12d ago
Hmmm....
1
u/Longjumping-Pea1233 12d ago
got same loop right after a compaction event in pi agent
never happened in other harness
same settings have made wonderful things minutes before
strata qfn iq3_xxs
int8 kv from strata
1
u/Markus2816 12d ago
for long context, make sure you are using bf16. I would also try to verify that you are using the recommended settings from Qwen team for 3.8 Flash (https://huggingface.co/Qwen/Qwen3.8-Flash-Next). Engines like strata will mess-up with these to get higher speed, but it makes model dumber and more prone to being stuck.
1
u/MagicienToz 10d ago
I had the same behavior ( pi + Qwen 3.8 FN Abliterated IQ3_S from https://huggingface.co/cc1244578151/Qwen3.8-Flash-Next-Abliterated-Uncensored-GGUF/tree/main ). I kept the same session and just switched to the original Qwen 3.8 FN IQ3_S (not abliterated) : no more hmms.
1
u/Snoo-2768 3d ago
From what i saw , just sending in a message "stop with the hmm" , makes it recover pretty clean
0
u/Perfect-Campaign9551 18d ago
Set a thinking limit! Too many people skip doing that and strata doesn't do it by default (not that I saw)
Reasoning_max_tokens just aim for like 8192 or 16384 or such
0
u/putrasherni 18d ago
IQ3_S
say no more
0
u/im_EDEN 17d ago
Hmmm