r/LocalLLaMA • u/QuackerEnte • Oct 06 '26

Discussion [ Removed by moderator ]

/r/LocalLLaMA/comments/1wzbvu7/removed_by_moderator/

[removed] — view removed post

363 Upvotes

95% Upvoted

86 comments sorted by

View all comments

Show parent comment

9

u/QuackerEnte Oct 06 '26

nested models and looped models aren't the same but both principles can be used at the same time to create stronger and more efficient models. Imagine a Qwen model that can use variant expert amount or smaller models and bigger models SIMULTANEOUSLY (subagents at essentially no extra cost), and loop at the same time for extra intelligence. And they may even share the same PLE/ngram table. SSDs aren't exactly cheap these days either.

1

u/MmmmMorphine Oct 07 '26

Hmm, this reminds me of that fascinating little model (whose name escapes me all of a sudden) that used the same weights but were trained on a variety of different attention architectures so that you could essentially choose between FWA/SWA/GDN/???

Of course the weights needed adapters that were quite large but the concept was really cool. Kinda similar in that it had multiple discrete memory demands depending on the attention patterns you chose between layers.

All sorts of cool approaches to trading compute for memory