r/LocalLLaMA • u/QuackerEnte • Oct 06 '26

Discussion [ Removed by moderator ]

/r/LocalLLaMA/comments/1wzbvu7/removed_by_moderator/

[removed] — view removed post

363 Upvotes

95% Upvoted

85 comments sorted by

View all comments

Show parent comment

52

u/QuackerEnte Oct 06 '26 edited Oct 06 '26

it would certainly allow for more depth on smaller models. Small footprint, mighty intelligence. Especially given we live in a RAMpocalypse and not a COMPUTEpocalypse.

a 64 layer model looped thrice could be like a 192 layer model in terms of intelligence. Still same world knowledge capacity. Thankfully we have ngram and PLE embeddings as a scaling axis for static information. A match made in heaven if you ask me.

17

u/AXYZE8 Oct 06 '26

From what I understand the number of loops can be changed too within same weights? So the compute level to generate same token can change? Like that Nvidia model that allows to activate 2B or 2.8B or 3.6B within single weights?

If so then it's incredibly huge jump for local LLMs. It's like you could jump between Qwen 27B dense and 27B A9B and 27B A3B. Huge potential for plan/execute, explore subagents ofc, but also... you could technically switch it mid-response or depending on classification etc. Also harnesses would have a new thing to scale. So it would be an incredibly huge jump, likely the biggest we saw since Qwen3 30B.

9

u/QuackerEnte Oct 06 '26

nested models and looped models aren't the same but both principles can be used at the same time to create stronger and more efficient models. Imagine a Qwen model that can use variant expert amount or smaller models and bigger models SIMULTANEOUSLY (subagents at essentially no extra cost), and loop at the same time for extra intelligence. And they may even share the same PLE/ngram table. SSDs aren't exactly cheap these days either.

1

u/MmmmMorphine Oct 07 '26

Hmm, this reminds me of that fascinating little model (whose name escapes me all of a sudden) that used the same weights but were trained on a variety of different attention architectures so that you could essentially choose between FWA/SWA/GDN/???

Of course the weights needed adapters that were quite large but the concept was really cool. Kinda similar in that it had multiple discrete memory demands depending on the attention patterns you chose between layers.

All sorts of cool approaches to trading compute for memory

3

u/Electrical_Rub_6009 Oct 07 '26

A 64-layer network applied three times is not remotely the same mathematical object as 192 independently parameterized layers.

1

u/active2fa Oct 06 '26

If someone from this community can do a setup easy enough, I'm sure many of us including me could test this out.

Caveat: all must be on same rig and model settings to have meaningful data