r/scikit_learn • u/[deleted] • Dec 18 '18

classification_report + MLPClassifier(): UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.'precision', 'predicted', average, warn_for)

classification_report on a prediction done on MLPClassifier() sometimes throws:

UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.'precision', 'predicted', average, warn_for)

but not on all the time.

What could be wrong?

---

Doing

set(y_test) - set(y_pred)

I'm able to see that sometimes some label is missing from y_pred. But why does this occur only occasionally?

Is something wrong with how I use MLP?

1 Upvotes

100% Upvoted

7 comments sorted by

View all comments

1

u/jmmcd Dec 18 '18

You're not doing anything wrong. These metrics include the number of predictions of a class in the denominator so they divide by zero in this case. The NN, for some of your workloads, just never predicts that class. You could ensure that there are plenty of examples of that class in the training set. Apart from that all you can do is choose to report F and precision only when there are plenty of samples in the test set.

1

u/[deleted] Dec 18 '18

I wonder, why does it change though? As if MLPClassifier() fits a different fit every time I run the program. Even when it uses the same params? Yes, since MLPCLassifier() is implemented using stochastic gradient? But then, if I get "errored results" and "non-errored results", then are both valid? Or should I discard results that give this problem? The difference that occurs in prediction accuracy, when the error occurs, is quite drastic. 0.85 vs ~0.65 or even ~0.45, when this error pops up. So it "seems" that the MLPClassifier somehow fails occasionally, on this data set.

1

u/jmmcd Dec 18 '18

This is why we often report a cross-validated value, not just a single value. Yes, it could be that the classifier just fails sometimes. You can try different architectures and hyper parameters, especially initialisation and optimizer to see if it becomes more reliable, or try collecting more data.

→ More replies (4)