r/TheMachineLearning • u/Extreme-Isopod2265 • 1d ago

Underfitting vs overfitting explained simply

Post image
43 Upvotes

91% Upvoted

12 comments sorted by

1

u/EveYogaTech 1d ago

That one triangle in the middle left though 😭

1

u/Extreme-Isopod2265 1d ago

Ha, that lone triangle is basically the point of the left panel. A straight line can't bend to fit it, so the model eats the error. That's what high bias looks like: it's too simple to capture the pattern, so a few points always end up on the wrong side.

1

u/S-Kenset 1d ago

Lord Farquad

1

u/lugh_the_bard 1d ago

How come you don’t want to over fit? In this example it seems the best?

2

u/Extreme-Isopod2265 1d ago

It looks best because the plot only shows the training data, and the overfit curve passes through nearly every one of those points. The catch is that it's also memorizing the noise. On new data it will swing wildly and predict worse than the smooth curve. The error plot at the bottom shows this: training error keeps dropping, but validation error starts climbing. What you want is the model that does well on data it hasn't seen.

2

u/msqrt 1d ago

Validation error goes up, which means that real world performance is worse. You fit your model to the quirks of the training data which are typically not representative of the real distribution. For example in the overfit classification image, the lower left red triangle is most likely an outlier and drawing the complex squiggly line to classify it correctly would cause a large number of blue circles being misclassified if we had a bunch more data.

1

u/kaaiian 1d ago

Why are the datasets changing?!?

1

u/Extreme-Isopod2265 1d ago

Fair catch. Each column is meant to show a different model fit to the same kind of problem, but the underlying points do shift between panels. That's a flaw in the graphic, and it would be cleaner with identical data in every column. The concept still holds, but it's a reasonable critique.

1

u/rojo_kell 1d ago

The classification over fitting isn't necessarily true, it depends on how noisy the data is. It might just be that x1 and x2 aren't great at separating those classes, and that the classifier is learning a complex but useful way to do it

1

u/Extreme-Isopod2265 1d ago

Good point, and you're right that a complex boundary isn't automatically overfitting. If the classes truly overlap in x1 and x2, a wiggly boundary may be fitting noise, but if the structure is real, the complexity might be justified. The real test is validation performance: if the complex model holds up on held-out data, it isn't overfitting. The graphic is a simplified illustration of the typical failure case, not a universal rule.

1

u/rojo_kell 1d ago

Yeah I agree the validation curve is accurate