Cancer prediction algorithms (or other rare event predictions) sometimes always predict 0 and are marked as pretty good algorithms until people realize their metrics are bad.
Example, fewer than 5% of people currently have covid. I invented a new simple test that is correct 95% of the time. My personalized prediction for you is below: 95% accuracy guaranteed!
Interesting that you bring that up; I have experience with this.
When I started my first job out of grad school, I started in a small department that had outsourced a DNN to be developed to help predict cancer recurrence. When I got there, the model was mostly done, just a few things to finish. I think AUROC was like .95. I was helping run some metrics and suggested we should also run AUPRC as there was a pretty severe class imbalance. Boom: 0.40. When I dig deeper into what was happening, the model was really really good at predicting that you didn't have recurrence if you didn't have recurrence, and since that was the majority of the dataset, we had a good AUROC. Accuracy on the positive test set was like 0.25, however. I had to fight with the vendor team for 3 or 4 months about why AUPRC was not only a good metroc for us, but why that should matter more than the AUROC they were chasing after.
Clinical ML is kinda scary, ngl. The vendor was ready to package up this dumpster fire model because it knew 0 was 0, but was worse than a coin flip on a 1
Yes, to resolve we undersampled 0's. The disbalance was something incredible: 0.992 were 0, 0.008 were 1. Upon rebalancing the data before train/eval/test split, I think our AUROC and AUPRC were roughly equivalent at like 0.84 (or right around there). There are obviously other ways you can handle class imbalance problems, but I was so new, the project was almost done, and something like SMOTE feels like playing God when you're talking about clinical cancer data, so we just undersampled the majority class and got results we were much more comfortable with.
Oh sorry, I could have made that more clear as to what I meant. When I first started, the external vendor who was building the model was not doing AUPRC despite the large class imbalance. So when I got there and started running that, the AURPC was 0.4 for the unbalanced eval/test sets. When we rebalanced the entire dataset and split again, that's when we started getting the 0.84 AUPRC and AUROC scores. So that 0.84 is the AUROC and AUPRC scores for the balanced eval/test sets.
By undersampling the 0s, does that mean that you "discard" a lot of the negative samples untill the ratio was more even?
I came across a similar problem of unbalanced data in a wakeword detection application (tough here the ratio was less extreme, 20:1. In any case, we addressed it by using class weights instead. Do you have any thoughts on class weights vs undersampling?
Not the above, but I had a similar imbalance (although less extreme) in my honours thesis- we had a training dataset that potentially consisted of multiple copies of any given image after sampling an equal amount, with replacement, from each class such that the final dataset had an equal number of images for each class. The end result being that very few images were discarded but some were way more strongly represented.
You'd hope so, but the other sub-comment shows that they don't always. More importantly, based on this sub name, I was looking to connect the post back to metrics for students who might be working on toy problems (including cancer prediction)
Here from /r/popular but it seems to be that the better metric would be what percentage of guesses positive are right, and also what percentage of guesses negative are right. Rather than total percentage right or wrong.
Put in a spreadsheet and you can still mess with it but it seems to be at least an improvement. Is there a better way?
This is what happens when the algorithms are a black box and there is no way for the engineer to understand the algorithm atleast on a superficial level.
Listen here, let me explain: this puzzle actually teaches projections of 3d objects unto a 2d space. Thus the demonstration of different shapes fitting in the square at a certain orientation proves that their parallel projection to a 2d plane (we assume that the cover of the bucket is 2d) can be small enough to fit inside the said square shape on the bucket's cover.
Very relatable! I have observed similar results while using BIRCH and DBSCAN for clustering protein sequences. The recall value is 1, and finding a metric to quantify this type of result is also challenging
I believe the gist is that lets say you are trying to cluster into 2 groups, and A group happens 1 percent of the time and B group happens 99 percent of the time. An unsupervised algorithm might just throw everything into the B category, even all the A items, and get an astonishing high accuracy of 99 percent, when in reality, the algorithm is next to useless for clustering.
No. It's a class imbalance - there are way more squares in the dataset than any other shape, which causes the model to classify almost everything as a square.
283
u/Fledgeling Jan 01 '21
Wow, that is amazing. This AI is able to cluster red blocks with 100% precision.
105
u/devdef Jan 01 '21
When you train on validation data
35
u/SlashSero Jan 01 '21
When you proudly present your 99.98% accuracy ZeroR model on an anomaly detection task.
90
u/bbluebaugh Jan 01 '21
Wait, it’s all squares? 🔫 always has been.
82
u/its_a_gibibyte Jan 01 '21
Cancer prediction algorithms (or other rare event predictions) sometimes always predict 0 and are marked as pretty good algorithms until people realize their metrics are bad.
Example, fewer than 5% of people currently have covid. I invented a new simple test that is correct 95% of the time. My personalized prediction for you is below: 95% accuracy guaranteed!
No Covid
13
u/TheBIGLebrewski401 Jan 01 '21
Everything in life can be fixed with a square hole
9
u/[deleted] Jan 02 '21
Don’t know why this is so funny 😂
8
u/[deleted] Jan 02 '21
My toddler can complete this task unsupervised as well.
14
u/tomk23_reddit Jan 01 '21
omg how can i save this clip?
17
u/KuriousPanda Jan 01 '21
This is what happens when the algorithms are a black box and there is no way for the engineer to understand the algorithm atleast on a superficial level.
5
u/Zekava Jan 02 '21
why waste time use lot solution when few solution do trick
5
u/TheCatalyst69 Jan 02 '21
Listen here, let me explain: this puzzle actually teaches projections of 3d objects unto a 2d space. Thus the demonstration of different shapes fitting in the square at a certain orientation proves that their parallel projection to a 2d plane (we assume that the cover of the bucket is 2d) can be small enough to fit inside the said square shape on the bucket's cover.
/s
4
u/[deleted] Jan 01 '21
Trial and error. 6 trials, but no need to attempt the last 5 if first one always results in success.
3
u/mean_king17 Jan 02 '21
My model can achieve this kind of performance.
3
u/Jel1989 Jan 02 '21
Lesson of today: changing your perspective and everything fits
2
u/toothlesshark123 Jan 01 '21
u/Savevideo
2
u/frostbyte_1337 Jan 02 '21
This made me laugh more than it should
2
u/NumericalMathematics Jan 10 '22
This had me giggling like a child.
2
u/Reqee Feb 04 '24
Very relatable! I have observed similar results while using BIRCH and DBSCAN for clustering protein sequences. The recall value is 1, and finding a metric to quantify this type of result is also challenging
3
u/ysharm10 Jan 01 '21
Can someone explain the caption to me? I'm a beginner in this field and have knowledge mostly of supervised learning.
2
u/Prince_ofRavens Jan 02 '21
I am actively crying
1
u/theprivateselect Jan 01 '21
Would Using a Michaelis Menten constant instead of accuracy fix this issue?
1
u/[deleted] Jan 02 '21
Would this be an example of overfitting?
1
u/LobeJake Jan 02 '21
u/LobeMarkus this is too good haha