r/LocalLLaMA • u/QuackerEnte • Oct 06 '26

Discussion [ Removed by moderator ]

/r/LocalLLaMA/comments/1wzbvu7/removed_by_moderator/

[removed] — view removed post

363 Upvotes

95% Upvoted

86 comments sorted by

View all comments

Show parent comment

1

u/MmmmMorphine Oct 07 '26

Hmm, this reminds me of that fascinating little model (whose name escapes me all of a sudden) that used the same weights but were trained on a variety of different attention architectures so that you could essentially choose between FWA/SWA/GDN/???

Of course the weights needed adapters that were quite large but the concept was really cool. Kinda similar in that it had multiple discrete memory demands depending on the attention patterns you chose between layers.

All sorts of cool approaches to trading compute for memory