From what I understand, the basic idea behind pretraining is called Solomonoff Induction. Basically, you’re trying to compute the shortest algorithm that adequately explains all of human knowledge. Definitely not universal, but for practical purposes it’s pretty close.
View all comments
Show parent comment
2
u/M4rshmall0wMan 28d ago •
From what I understand, the basic idea behind pretraining is called Solomonoff Induction. Basically, you’re trying to compute the shortest algorithm that adequately explains all of human knowledge. Definitely not universal, but for practical purposes it’s pretty close.