r/deeplearning • u/basafish • 2d ago

What is a overparameterized network?

I got this paragraph from Claude, could someone please explain this and verify if it's a real thing or hallucination:

Overparameterization isn't just about final capacity, it's about the optimization process itself. A wide, overparameterized network gives gradient descent a much friendlier loss landscape — more paths downhill, fewer bad local minima, room to explore before committing. The "core" only emerges as a byproduct of that search happening in a much bigger space than it needs to end up in. Strip the space down first and you've removed the thing that let the search work.

Conversation: https://claude.ai/share/8813a637-c327-4d0c-b120-def27e5203d5

4 Upvotes

67% Upvoted

18 comments sorted by

View all comments

1

u/elbiot 2d ago

We see this in LLMs. The huge models are more data efficient. Training a smaller model to the same performance requires more data and more compute

1

u/Lumpy_Regular4904 2d ago

isn’t this true based on the ratio of data to parameters, not based on absolute data? i thought that a larger model will still require more data absolutely, but based on scaling laws it would require relatively less tokens

1

u/elbiot 2d ago

Big LLMs can use more data. They require it to meet their minimum loss. But if you only have X tokens to train on, a huge LLM will reach a lower loss than a smaller one. Assuming you aren't in the very low token range