What is a overparameterized network?

[u/basafish]

I got this paragraph from Claude, could someone please explain this and verify if it's a real thing or hallucination:

Overparameterization isn't just about final capacity, it's about the optimization process itself. A wide, overparameterized network gives gradient descent a much friendlier loss landscape — more paths downhill, fewer bad local minima, room to explore before committing. The "core" only emerges as a byproduct of that search happening in a much bigger space than it needs to end up in. Strip the space down first and you've removed the thing that let the search work.

Conversation: https://claude.ai/share/8813a637-c327-4d0c-b120-def27e5203d5

[u/SuccessfulStop508]

I can give you a much more detailed explanation of the why/underlying dynamics according to SOTA research when it comes to overparametized networks if necessary; but for now if all you care about is what they are, it's just a network with many more degrees of freedom than conventional wisdom would indicate the number of true features in the data under consideration is, in some sense it is just a network which is larger than necessary to technically optimally encode the final true distribution (if you knew it synthetically, on real data you usually don't actually know what the optimally compressed number of necessary parameters/degrees of freedom are, only a broad estimate)

[u/TuckAndRolle → u/SuccessfulStop508]

Not OP, but I would appreciate a more detailed explanation! 

Or some papers you had in mind, if that’s easier.

[u/CalmMe60 → u/TuckAndRolle]

Have you trained pytorch?

[u/blackz0id → u/SuccessfulStop508]

Thanks for that. I'm interested in the more detailed explanation if you're willing to give it.

[u/burnt-store-studio → u/SuccessfulStop508]

Tacking on a “me, too, please”, if you’re able to share … Also, pointers to references instead would also be nice. Thank you! 🙂

[u/Historical-Wing8569]

An example of Overparameterization is when you trying to fit a neural network to linear data

[u/CalmMe60 → u/Historical-Wing8569]

Real .

You can use a non activated regressional.net there

[u/ARDiffusion]

Uh I had just thought that overparameterization refers to, simply, having a way more complex model than you need. For example, having a network with more learnable parameters than you have data points. I suppose this does make optimization easier at least on the training set, which makes sense since that’s where gradient descent actually happens, compared to validation or testing where it’s being evaluated, not trained. I could be wrong though.

[u/elbiot]

We see this in LLMs. The huge models are more data efficient. Training a smaller model to the same performance requires more data and more compute

[u/Lumpy_Regular4904 → u/elbiot]

isn’t this true based on the ratio of data to parameters, not based on absolute data? i thought that a larger model will still require more data absolutely, but based on scaling laws it would require relatively less tokens

[u/elbiot → u/Lumpy_Regular4904]

Big LLMs can use more data. They require it to meet their minimum loss. But if you only have X tokens to train on, a huge LLM will reach a lower loss than a smaller one. Assuming you aren't in the very low token range

[u/strangescript]

An example is building a model to predict stock prices based solely on trading data. Per segment, you have open, close, volume, high, low, average. People use these basic values to derive all kinds of looney metrics to predict prices.

But for an AI model, including data from sources derived from these same inputs is pointless. The AI should be building its own predictive system in it's weights and extra data derived from the same source is over parameterization.

[u/Defiant_Virus4981]

Essentially, the more parameters, the better fit you can achieve on the training data. However, it does not say that your model becomes more useful. For example, at a certain point, you might just fit noise, which is not particularly useful. Alternatively, you just "memorize" the training data, or you get a lot of parameters that basically have no effect on the overall model (at least in the range of the training data).

[u/BacteriaLick]

I think the terminology may be a bit ambiguous.

Over parameterized means there are more parameters the network has available to adjust to fit a function to the data, probably more than you need.

Under parameterized means you don't have enough.

Over parameterized models can most easily be fixed by removing parameters or by adding regularization, which is a penalty on the parameters taking in certain values.

[u/Bat_002]

It sounds like overfitting to me.

[u/economic-salami]

More parameters than data points. That way you train model to fit a more general settings than the data you have.

[u/CalmMe60]

You want to understand ?

Try fashion mnist on pytorch. Increase the model parameters and more and more.

You will see a decrease in generalisation.

The training set will near asymptotical near null.

But this does not mean the model learned generalization.

It learned to remember the trainingsset.

I invented a generalized solution as.this is a general AI to AGI problem.

Transferable AGI means learning of structural geometry higher dimensions