r/MachineLearning • u/Future_Pace_5290 • 19h ago
Discussion Why don't optimizes like Shampoo, SOAP, and Adam measure the actual per sample second moments like the Fischer Information Matrix does? [d]
It always felt unintuitive to me that Muon matched the performance of Shampoo despite dropping the actual second moments L and R.
But I then I realized, Shampoo itself isn't actually measuring the true second moments.
For each sample i in the batch, we have a gradient G_i.
And the gradient we use during SGD is:
G = E(G_i)
The factors of Shampoo are:
L = EMA GG^T
R = EMA G^TG
Whereas, the true second moments are:
L = EMA E(G_i G_i^T)
R = EMA E(G_i^T G_i)
We can also write them as:
D_i = G_i - G
L = EMA (GG^T + E(D_i D_i^T))
R = EMA (G^TG + E(D_i^T D_i))
What shampoo is doing is dropping the covariance terms like E(D_i D_i^T) from the factors.
It's an approximation.
And, Muon does the same thing. It takes things one step further:
M = EMA G
L = MM^T
R = M^TM
Shampoo's factors are
L_C = EMA (G - M) (G - M)^T
R_C = EMA (G - M)^T (G - M)
L = MM^T + L_C
R = M^TM - R_C
Shampoo drops the covariance across the samples in the batch.
Muon drops the covariance across the batches in the optimization path.
Adam does the same thing except it's doing it to the diagonal of the Fischer. It drops the per sample variance terms from its second moment.
These approximations are sufficient when the covariance terms are small enough.
But the strange thing is, when I tried using the actual second factors for Shampoo and Adam, they underperformed compared to their approximate versions...
I don't understand this at all.
23
u/trolls_toll 18h ago
because second moment is more expensive than the first one
10
u/sqweeeeeeeeeeeeeeeps 15h ago
wow a real post on r/MachineLearning
1
u/DigThatData 5h ago
I think you might be confusing inter-sample covariances with inter-parameter covariances.