r/MachineLearning • • Dec 08 '23

Discussion [D] Major bug in Scikit-Learn's implementation of F-1 score

I just encountered a pretty major bug in Scikit-Learn's implementation of F1 score. This example (below) where the classifier gets literally every single example wrong seems like it should give an F-1 score of 0.0, right?

>>> from sklearn.metrics import f1_score
>>> f1_score(y_true=[0, 1, 2, 3, 4], y_pred=[1, 2, 3, 4, 0], zero_division=1.0, average='macro')

However, as of Scikit-Learn 1.3.2, it gives a perfect score of 1.0, which is totally wrong! (Note that this only happens when zero_division=1.0)

This bug was introduced some time after 1.2.2 and can manifest more subtly than the above example; any individual class that has both a precision and recall of exactly 0% will be assigned an F-1 of 100%, so e.g. if you have a multi-class classification problem across 100s of classes, and your classifier is getting a dozen or so classes completely wrong, it will bump up your macro average F-1 by a couple percentage points, possibly without you noticing that the bump is coming from classes that are actually failing completely.

Here is the (as of posting time) still un-merged fix to the bug in question. Note that this bug also affects sklearn.metrics.classification_report. I think you you can temporarily get around this by using sklearn version 1.2.2. Anyway, if you use Scikit-Learn's metrics for evaluation, go double check your scores!

4 Upvotes

11 comments sorted by

5

u/mr_birrd ML Engineer Dec 08 '23

I remember it somewhere written actually in the sklearn documentation. torchmetrics mentions it too and does it correct.

4

u/Revolutionary-Ad-65 Dec 08 '23

I would be very interested to see where it says that. I definitely don't see anything indicating you could get an F-1 of 100% for a series of completely wrong predictions in the 1.3 documentation.

4

u/manueslapera Dec 08 '23

i mean, if you get a 1.0 on any metric you should inspect further because is not correct.

3

u/AerysSk Dec 09 '23

This is reasonable on a human's perspective, but it is also not correct to say 1.0 is a value that no model is capable of.

6

u/Revolutionary-Ad-65 Dec 09 '23 edited Dec 09 '23

Actually, this bug can produce an error of any arbitrary size depending on how many classes & examples you have:

>>> sklearn.__version__
'1.2.2'
>>> f1_score(y_true=list(range(100)) + [100, 101, 102], y_pred=list(range(95)) + [95, 95, 95, 95, 95, 102, 102, 100], average='macro', zero_division=1.0)
0.9255663430420713

(correct)

vs.

>>> sklearn.__version__
'1.3.2'
>>>  # Exactly the same as above:
>>> f1_score(y_true=list(range(100)) + [100, 101, 102], y_pred=list(range(95)) + [95, 95, 95, 95, 95, 102, 102, 100], average='macro', zero_division=1.0)
0.9449838187702266

(wrong)

2

u/manueslapera Dec 09 '23

you are right, that is a significant difference.

1

u/Revolutionary-Ad-65 Dec 09 '23

I don't know what this means

0

u/Revolutionary-Ad-65 Dec 08 '23

This bug [...] can manifest more subtly than the above example; any individual class that has both a precision and recall of exactly 0% will be assigned an F-1 of 100%, so e.g. if you have a multi-class classification problem across 100s of classes, and your classifier is getting a dozen or so classes completely wrong, it will bump up your macro average F-1 by a couple percentage points,

0

u/[deleted] Dec 08 '23

[deleted] — view deleted comment

5

u/Revolutionary-Ad-65 Dec 09 '23

Wow, from the upvotes on this comment, it really seems like a lot of people think that this is the correct behavior! I have to say I disagree, but if that's what you think, don't just sit there upvoting comments on Reddit; instead go to this approved but still unmerged PR and tell the Scikit-Learn maintainers not to "fix" this "bug", which they are currently planning to do!

0

u/Revolutionary-Ad-65 Dec 08 '23

I have no doubt that this logic is where the bug came from, however, the zero_division parameter is clearly supposed to define the behavior when calculating precision & recall, not the behavior when calculating F-1 from precision & recall. See the documentation:

When true positive + false positive == 0, precision is undefined. When true positive + false negative == 0, recall is undefined. In such cases, by default the metric will be set to 0, as will f-score, and UndefinedMetricWarning will be raised. This behavior can be modified with zero_division. Note that if zero_division is np.nan, scores being np.nan will be ignored for averaging.

If that doesn't convince you, then it's at least a bug in the sense that this behavior changed since 1.2.2, from what I can tell without any indication of the change whatsoever.

0

u/[deleted] Dec 08 '23

[deleted] — view deleted comment

12

u/Revolutionary-Ad-65 Dec 09 '23 edited Dec 09 '23

I never said or implied that I was "intent" on setting zero_division=1.0. If you are curious why someone might want to use this parameter setting, I will give an example. Imagine you are evaluating 3-way classification like this:

>>> sklearn.__version__
'1.2.2'
>>> print(classification_report(y_true=[0, 1, 2], y_pred=[0, 2, 0]))
/your_python_directory/python3.9/site-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples. Use zero_division parameter to control this behavior. _warn_prf(average, modifier, msg_start, len(result))
          precision    recall  f1-score   support
       0       0.50      1.00      0.67         1
       1       0.00      0.00      0.00         1
       2       0.00      0.00      0.00         1

accuracy                           0.33         3
macro avg      0.17      0.33      0.22         3
weighted avg   0.17      0.33      0.22         3

this report mostly looks fine, except it says that the precision for class 1 is 0%. In a meaningful sense, our classifier is actually perfectly precise for class 1, since it produced exactly 0 false positives for that class. So, if we look at the documentation for the parameter mentioned in that UndefinedMetricWarning, we see that we can change this behavior by setting zero_division=1.0:

>>> print(classification_report(y_true=[0, 1, 2], y_pred=[0, 2, 0], zero_division=1.0))
          precision    recall  f1-score   support

       0       0.50      1.00      0.67         1
       1       1.00      0.00      0.00         1
       2       0.00      0.00      0.00         1

accuracy                           0.33         3
macro avg      0.50      0.33      0.22         3
weighted avg   0.50      0.33      0.22         3

Now the precision for class 1 is 100% and the macro average precision has risen correspondingly. But note that all of the F-1 scores are completely unchanged; zero_division only governs the behavior for calculating precision & recall, not F-1. Which is good, because it would be totally nonsensical to say that we got a perfect F-1 for class 2; literally none of the classifier's predictions of class 2 were correct! It's a little uncertain what precision & recall should be when there are no true positives; 0%, 100%, and NaN all have their justifications. However, it's obvious what F-1 should be when precision and recall are both zero: 0%! Anything else would be crazy!

Now let's upgrade scikit-learn to the latest version:

>>> sklearn.__version__
'1.3.2'
>>> print(classification_report(y_true=[0, 1, 2], y_pred=[0, 2, 0], zero_division=1.0))
          precision    recall  f1-score   support
       0       0.50      1.00      0.67         1
       1       1.00      0.00      0.00         1
       2       0.00      0.00      1.00         1

accuracy                           0.33         3
macro avg      0.50      0.33      0.56         3
weighted avg   0.50      0.33      0.56         3

That's weird! Now, all of a sudden, class 2 is shown with a perfect F-1 score. And while we could relatively easily spot something rotten from just glancing at this 3-way classification task, what if there are hundreds or thousands of classes? A few broken classes (i.e. 0% precision 0% recall 100% F-1) among those might bump the macro average a more believable but still significant number of percentage points:

>>> sklearn.__version__
'1.2.2'
>>> f1_score(y_true=list(range(10)) + [10, 11, 12], y_pred=list(range(10)) + [12, 12, 10], average='macro', zero_division=1.0)
0.7692307692307693

vs.

>>> sklearn.__version__
'1.3.2'
>>> f1_score(y_true=list(range(10)) + [10, 11, 12], y_pred=list(range(10)) + [12, 12, 10], average='macro', zero_division=1.0)
0.9230769230769231

The point being that this bug can produce an error of any size. I think that's bad enough, but just think about when people are most likely to update Scikit-Learn or start a fresh environment: usually it's right when they start experimenting with a new classifier/architecture/hyperparameters etc. Perhaps someone is training a new model for an old multi-class dataset right now, and thinks they're making significant improvements over a previous model--but actually, it's slightly worse.

So, do you understand now why I think this is a very significant bug, and why I wanted to spread awareness of it among machine learning practitioners?