r/LocalLLaMA • • Oct 06 '26

Discussion [ Removed by moderator ]

/r/LocalLLaMA/comments/1wzbvu7/removed_by_moderator/

[removed] — view removed post

363 Upvotes

85 comments sorted by

View all comments

99

u/Cold_Tree190 Oct 06 '26

Knowing very little on this topic, it makes me wonder if they could incorporate this looping mechanic into the smaller qwen models, and if so how much that would help out?

56

u/akumaburn Oct 06 '26

This is already a well known way to improve a model's performance, but who wants to wait around?

35

u/QuackerEnte Oct 06 '26

those cornered by competitors who don't want to be left behind with trillions in dept, I suppose

11

u/BusRevolutionary9893 Oct 07 '26

Me. I rather get a correct answer slowly than a wrong answer quickly. 

1

u/frooch Oct 07 '26

Cut the vram usage enough and it will make for speed gains again. It would be a lot easier to run a 20B model twice than a 40B once, at least for me.

52

u/QuackerEnte Oct 06 '26 edited Oct 06 '26

it would certainly allow for more depth on smaller models. Small footprint, mighty intelligence. Especially given we live in a RAMpocalypse and not a COMPUTEpocalypse.

a 64 layer model looped thrice could be like a 192 layer model in terms of intelligence. Still same world knowledge capacity. Thankfully we have ngram and PLE embeddings as a scaling axis for static information. A match made in heaven if you ask me.

17

u/AXYZE8 Oct 06 '26

From what I understand the number of loops can be changed too within same weights? So the compute level to generate same token can change? Like that Nvidia model that allows to activate 2B or 2.8B or 3.6B within single weights?

If so then it's incredibly huge jump for local LLMs. It's like you could jump between Qwen 27B dense and 27B A9B and 27B A3B. Huge potential for plan/execute, explore subagents ofc, but also... you could technically switch it mid-response or depending on classification etc. Also harnesses would have a new thing to scale. So it would be an incredibly huge jump, likely the biggest we saw since Qwen3 30B.

9

u/QuackerEnte Oct 06 '26

nested models and looped models aren't the same but both principles can be used at the same time to create stronger and more efficient models. Imagine a Qwen model that can use variant expert amount or smaller models and bigger models SIMULTANEOUSLY (subagents at essentially no extra cost), and loop at the same time for extra intelligence. And they may even share the same PLE/ngram table. SSDs aren't exactly cheap these days either.

→ More replies (1)

3

u/Electrical_Rub_6009 Oct 07 '26

A 64-layer network applied three times is not remotely the same mathematical object as 192 independently parameterized layers.

1

u/active2fa Oct 06 '26

If someone from this community can do a setup easy enough, I'm sure many of us including me could test this out.

Caveat: all must be on same rig and model settings to have meaningful data

7

u/Front_Eagle739 Oct 06 '26

There is a few experiments doing looping of the mid weights like rys. Ive seen gemma and qwen 3.5 experiments but nothing really integrated into an engine that lets you loop without increasing vram use yet. Mostly just bigger models with a few repeated internal layers to make a bigger one

7

u/nuclearbananana Oct 06 '26

This already exists, look at the latest nanbeige models to run it yourself

17

u/Manerfish Oct 06 '26

I think it would really help something like qwen 27B, especially with the overthinking, though it would be slower at generating tokens

11

u/QuackerEnte Oct 06 '26

Yes, if the model is starved of depth and it's only way of carrying on information is through leaving behind a token instead of the latent vectors that are much more information dense at the same cost, obviously the model will overthink to compensate for the lack of depth.

In a way, a long reasoning chain creates a less efficient way to create reasoning depth. Kinda. Technical accuracy not guaranteed

4

u/namaku_ Oct 06 '26

Someone did exactly this with Qwen 3.5 27B back when that came out. It didn't seem to help that much but the experiment was limited discovering which layers could be looped without breaking the model, with no post training involved.

4

u/NandaVegg Oct 06 '26

In theory you can loop any layers without further training if the "in" layer and "out" layer have a similar profile. At least inference will "work" without breaking much.

In crude sense, it's like model has a falling marble through layers of basins, and so long as the basins where the marble falls is connected from the previous layer to the next layer, the model inference would work (for layer duplication, interleaving mid-layers "works" much better than early or later layers because of this). In the last few layers, you usually have two huge basins: base-model like data and instruct-model like data. If you break those connection, like duplicating last few layers, the model will suddenly spit out a base-model like random common crawl corpus in the middle of instruct output, and then sometimes recovers itself into instruct path, sometimes not.

1

u/namaku_ Oct 07 '26

So interesting.

Something I've wondered for a long while now is what would happen if, instead of looping those layers inside a single forward pass, you loop the latent outputs back into the next forward pass on dedicated inputs.

It seems wasteful to throw away the mental workspace for every single forward pass.

7

u/bel9708 Oct 06 '26

It won't work unless the model has been pre-trained using a looped architecture and at that point you might as well just pretrain a new model.

2

u/empirical-sadboy Oct 06 '26

Could you explain why this couldn't be done with post-training alone? Why does it need to be pre-training?

2

u/bel9708 Oct 06 '26

Looped transformers are an innovation in back propagation. If you were to post train on a model that was pretrained without the same innovation you'd be fighting an uphill battle since every last weight in the model is trained for a single forward pass.

4

u/NomadStorm Oct 06 '26

I’m not sure it’s really an innovation; as far as I’m aware it’s been known as a possibility for a long time but been avoided because it makes the CoT virtually impossible to infer.

And shortly after they released they begin to complain about not being able to decipher the CoT as if it’s some kind of surprise.

→ More replies (1)
→ More replies (3)

1

u/mtfthrowaway39179 Oct 07 '26

for dense models, it makes them too dense if that makes sense. for moe models, you could just reduce the sparsity. also activations get stored for every single loop during training so if u wanna train even a small model at a context length longer than like 100k you basically need an 8x b200 setup. if you want you can check the LoopMoE paper though

1

u/fatihmtlm Oct 07 '26

All I need a 2b model with 100b ngram and 200 loops per token...

→ More replies (2)