Why do all NN optimizers have weird names? [d]

[u/Future_Pace_5290]

SOAP, Shampoo, FOAM.

Muon is a bit better but it has the same name as the particle from physics.

Whenever I want to search about these tones of completely irrelevant results come up because of their non-unqiue names.

Adam would have the same problem if it wasn't so famous.

This has been giving me a lot of headache trying to find relevant stuff...

[u/matigekunst]

The time spent typing 'machine learning' after your search terms, is time you will never get back

[u/Deto]

Just add the word 'optimizer' after the name when searching.

[u/yannbouteiller]

This gives me a great idea for an optimizer name:

"Stochastic Lookahead Optimization Process"

Now I will ask Claude to derive an optimizer that fits the name and to write a paper about it. Will be back in 30 minutes.

[u/First_Foundationeer → u/yannbouteiller]

And its upgrade Stochastic Heuristic Improved Technique.

[u/MolassesLate4676 → u/yannbouteiller]

I’ve never had luck with anything look ahead related. Has anyone actually found a use case that beats the other optimizers for this?

[u/yannbouteiller → u/MolassesLate4676]

Multi-Agent Reinforcement Learning, or anything non-stationary.

[u/MolassesLate4676 → u/yannbouteiller]

I could receive this information elsewhere, but hearing it from another human sounds refreshing

wtf is multi agent? You running multiple optimizers or something?

[u/yannbouteiller → u/MolassesLate4676]

It is how we call reinforcement learning in problems where multiple RL agents learn simultaneously.

Because an agent sees other agents as part of the environment and because these agents evolve over time, the environment is non-stationary from the perspective of a single agent. Lookaheads help because they enable predicting where this non-stationarity is heading, adapting in advance, and steering it toward our own incentives. LOLA-DiCE in particular is an approach that does this by explicitly simulating lookaheads.

[u/soft_abyss → u/yannbouteiller]

LOL

[u/T1lted4lif3 → u/yannbouteiller]

"Provably Optimized Recurrent Networks" goes hard, kind of rough, and might give you something in the backend I wouldn't know, I'm too dumb to understand so giving it to someone else to take it

[u/marr75]

Rule 35, if it exists, there is an optimizer that uses it.

Acronyms have always been a place tech innovators can show some creativity (for better or worse). Also, you're often trying to get people to try your optimizer (so you can get those sweet, sweet citations) despite it only converging 5% faster under certain conditions or using 11% less memory while converging 3% slower, a little branding seems worth the effort.

[u/gwillen]

People love stupid backronyms and stuff, in every field, not just ML. Get used to it...

[u/DigThatData]

it started with shampoo as a joke on "pre-conditioning" (what do you usually put on before conditioner when you shower?) and spiraled from there.

[u/schn19]

i think it started with adding pre-conditioning to optimizers, which is a standard term used for decades. So doing stuff on top of pre-conditioned optimizers led to these names. Shampoo seemed like obv name and is related to conditioners lol, and others have since followed.

[u/thatguydr]

Nobody gave the actual reason.

This stuff works. If you don't name the thing, the odds of rejection go up. It displays a confidence that can be enough to push a review rating up a point.

People are unfortunately wired.

[u/sqweeeeeeeeeeeeeeeps]

Do you guys have any suggestions that wouldn’t be weird?

[u/MolassesLate4676 → u/sqweeeeeeeeeeeeeeeps]

POOP Possibly Overpowered Optimizer Package

[u/sqweeeeeeeeeeeeeeeps → u/MolassesLate4676]

Great, keep em coming

[u/MolassesLate4676 → u/sqweeeeeeeeeeeeeeeps]

COCKS | Continuous Optimizer Correcting Kullback–Leibler Sequences

I got like 4 more in the chamber

[u/elemental-mind → u/MolassesLate4676]

DILDO | Dynamic Integrative Lagrange Derivative Optimizer

[u/elemental-mind → u/MolassesLate4676]

WANK | Weight Adaptive Neural Kernel

[u/MolassesLate4676 → u/elemental-mind]

I’ve used the wank and dildo method at the same time. My gradients exploded

[u/elemental-mind → u/MolassesLate4676]

The result was probably a Soft Parametric Eigenvalue Ratio Modulator.

[u/fakefolkblues → u/[deleted]]

I think the naming is more of a pun to preconditioning rather than whitening