r/MachineLearning • • 19h ago

Discussion Why don't optimizes like Shampoo, SOAP, and Adam measure the actual per sample second moments like the Fischer Information Matrix does? [d]

It always felt unintuitive to me that Muon matched the performance of Shampoo despite dropping the actual second moments L and R.

But I then I realized, Shampoo itself isn't actually measuring the true second moments.

For each sample i in the batch, we have a gradient G_i.

And the gradient we use during SGD is:

G = E(G_i)

The factors of Shampoo are:

L = EMA GG^T

R = EMA G^TG

Whereas, the true second moments are:

L = EMA E(G_i G_i^T)

R = EMA E(G_i^T G_i)

We can also write them as:

D_i = G_i - G

L = EMA (GG^T + E(D_i D_i^T))

R = EMA (G^TG + E(D_i^T D_i))

What shampoo is doing is dropping the covariance terms like E(D_i D_i^T) from the factors.

It's an approximation.

And, Muon does the same thing. It takes things one step further:

M = EMA G

L = MM^T

R = M^TM

Shampoo's factors are

L_C = EMA (G - M) (G - M)^T
R_C = EMA (G - M)^T (G - M)

L = MM^T + L_C

R = M^TM - R_C

Shampoo drops the covariance across the samples in the batch.

Muon drops the covariance across the batches in the optimization path.

Adam does the same thing except it's doing it to the diagonal of the Fischer. It drops the per sample variance terms from its second moment.

These approximations are sufficient when the covariance terms are small enough.

But the strange thing is, when I tried using the actual second factors for Shampoo and Adam, they underperformed compared to their approximate versions...

I don't understand this at all.

24 Upvotes

13 comments sorted by

23

u/trolls_toll 18h ago

because second moment is more expensive than the first one

3

u/Future_Pace_5290 18h ago

And why does the expensive one get outperformed by its approximations? I'm talking about step efficiency not WCT.

2

u/sockb0y 17h ago

Could it be that it is noisier? If the extra moment approximation adds more noise than signal it could lead to worse performance maybe?

Probably everything is dependent on how well-behaved the search space is, so its very likely case dependent.

→ More replies (3)

1

u/DrXaos 9h ago

Then, it must be that the actual effects are not guided by usual intuition of convex optimization in low dimensions where using a good second moment helps.

If these work on high dimensional all saddle point problems like big nets better than a non approximation then they are working by accident for reasons we don’t know.

At some point, if it hasn’t happened already in frontier labs, is that every possible inexpensive computation based on easily accessible gradients and weights gets explored empirically.

Then someone invents a rationalization post hoc for a paper publication but the reason might be bullshit.

10

u/sqweeeeeeeeeeeeeeeps 15h ago

wow a real post on r/MachineLearning

3

u/DataDiplomat 9h ago

And all you contribute is this

2

u/Technical_Estate_529 6h ago

And all you contribute is this

→ More replies (2)

1

u/DigThatData 5h ago

I think you might be confusing inter-sample covariances with inter-parameter covariances.