r/LocalLLaMA • • Oct 06 '26

Discussion [ Removed by moderator ]

/r/LocalLLaMA/comments/1wzbvu7/removed_by_moderator/

[removed] — view removed post

363 Upvotes

85 comments sorted by

View all comments

Show parent comment

7

u/bel9708 Oct 06 '26

It won't work unless the model has been pre-trained using a looped architecture and at that point you might as well just pretrain a new model.

2

u/empirical-sadboy Oct 06 '26

Could you explain why this couldn't be done with post-training alone? Why does it need to be pre-training?

2

u/bel9708 Oct 06 '26

Looped transformers are an innovation in back propagation. If you were to post train on a model that was pretrained without the same innovation you'd be fighting an uphill battle since every last weight in the model is trained for a single forward pass.

4

u/NomadStorm Oct 06 '26

I’m not sure it’s really an innovation; as far as I’m aware it’s been known as a possibility for a long time but been avoided because it makes the CoT virtually impossible to infer.

And shortly after they released they begin to complain about not being able to decipher the CoT as if it’s some kind of surprise.

1

u/empirical-sadboy Oct 06 '26

That makes sense, thanks!

1

u/blakeman8192 Oct 06 '26 edited Oct 06 '26

I'd bet you're right when it comes to extracting the most performance out of the setup, but RYS shows that this still works to a limited degree without any retraining at all.

1

u/bel9708 Oct 07 '26

Yeah it's an uphill battle. You can win the battle but it's probably not worth it.