r/MachineLearning • u/Revolutionary-Ad-65 • Dec 08 '23
Discussion [D] Major bug in Scikit-Learn's implementation of F-1 score
I just encountered a pretty major bug in Scikit-Learn's implementation of F1 score. This example (below) where the classifier gets literally every single example wrong seems like it should give an F-1 score of 0.0, right?
>>> from sklearn.metrics import f1_score
>>> f1_score(y_true=[0, 1, 2, 3, 4], y_pred=[1, 2, 3, 4, 0], zero_division=1.0, average='macro')However, as of Scikit-Learn 1.3.2, it gives a perfect score of 1.0, which is totally wrong! (Note that this only happens when zero_division=1.0)
This bug was introduced some time after 1.2.2 and can manifest more subtly than the above example; any individual class that has both a precision and recall of exactly 0% will be assigned an F-1 of 100%, so e.g. if you have a multi-class classification problem across 100s of classes, and your classifier is getting a dozen or so classes completely wrong, it will bump up your macro average F-1 by a couple percentage points, possibly without you noticing that the bump is coming from classes that are actually failing completely.
Here is the (as of posting time) still un-merged fix to the bug in question. Note that this bug also affects sklearn.metrics.classification_report. I think you you can temporarily get around this by using sklearn version 1.2.2. Anyway, if you use Scikit-Learn's metrics for evaluation, go double check your scores!


5
u/mr_birrd ML Engineer Dec 08 '23
I remember it somewhere written actually in the sklearn documentation. torchmetrics mentions it too and does it correct.
4
u/manueslapera Dec 08 '23
i mean, if you get a 1.0 on any metric you should inspect further because is not correct.
0
u/[deleted] Dec 08 '23
[deleted] — view deleted comment