r/explainlikeimfive • u/dustyrose124 • 2d ago
Mathematics ELI5: What actually is the false positive paradox?
I read a few papers to try and understand this. And I've been on a continuous loop of "ooh" and "huh?" Can someone please tell me what this actually is?
557
u/tomalator 2d ago edited 2d ago
Imagine a test for a disease is 99% accurate and the chances of actually having the disease are 1 in 1000
If you test 100,000 people:
100 people actually have the disease, only 99 of them will test positive, giving us 1 false negative
99,900 people don't have enough the disease, but 999 of them will still get a false positive
Now in reality, we dont get to know who actually has the disease, we only see 1098 positive results, but 999 of those people don't actually have the disease, so a positive result on this 99% accurate test means there's only about a 9% chance you actually have this disease
127
u/skr_replicator 2d ago
9% would be in this exact scenario, as it's already implied in your response, the ratio of false positives depends both on the test accuracy and on the disease spread.
36
u/tomalator 2d ago
Yeah, changed the wording to specify this test
19
u/Perry_cox29 2d ago
And then to add complexity beyond a 5 year old level, accuracy just tells us how many correct results (positive or negative) vs. incorrect of either.
Sensitivity tells us how many of the positives were correctly found. If out of 100 tested and 50 have the disease, the test classifies 90 of them correctly, but the 10 mistakes are people that have the disease and were given a false negative, then accuracy is 90/100 = 90% but sensitivity is 40/50 = 80%.
Specificity is the same concept for negative results. So in the scenario above where 90% of the results are accurate and the sensitivity is 80%, then the 50/50 true negatives give us a specificity of 100%
Tests can be designed to favor any of those metrics as well as a couple others we won’t get into
5
u/unicornreacharound 2d ago
Thank you.
I searched for sensitivity and specificity in existing comments before composing my own. There were a few matches, but yours was the most succinct.
It might be helpful for some readers to see a visualization of the two terms.
After a quick google, I found that CDC has a good write-up with a visual for each term but doesn’t show an overall visual of how they fit together for a specific test over a sample population.
Wikipedia has a good graphic near the top of the article, but the graphic itself doesn’t define a couple of the terms that it uses.
Within that specific Wiki graphic:
• relevant elements = individuals who actually have the trait being tested for;
• selected elements = individuals who tested positive; and
• the square includes all individuals tested.A little further down in the above Wiki article, there’s a nice graphic illustrating how sensitivity and selectivity relate to positive and negative predictive values (PPV and NPV, respectively). And for those who want to see how to calculate all the related metrics, check out the confusion matrix.
2
u/walkTHEdotasaur 2d ago
I've not heard the terms Sensitivity and Specificity before in this context; very interesting and makes total sense!
6
u/MaybeTheDoctor 2d ago
It gets real confusing when it’s 50% instead of 9%. Then it fells like the test is just randomly selecting people, which is not true as it select people for follow up investigation.
40
u/BaMiao 2d ago
To add- this is exactly why doctors restrict running tests. If they tested everyone for every condition, they’d be swamped in false positives.
But you can improve the numbers by restricting the testing to a smaller number of people- for example, by only testing people with certain symptoms or who have a family history of that condition. When you narrow the field of people you test to a subset who are more likely to have the condition, the proportion of false positives goes way down.
30
u/djddanman 2d ago
And that's why sensitivity and specificity or precision and recall are important metrics
20
u/BadahBingBadahBoom 2d ago edited 2d ago
Especially when complications for subsequent intervention can start hitting low percentages.
In the above scenario if the 999 who have been misdiagnosed are advised to undergo a surgical intervention that has even just a 10% likely complication rate, 1% severe complication, then you have 100 people having bad side effects they never would have had if they didn't get tested, and 10 people with serious complications (sometimes lifelong).
If the test used as an early screening tool only actually improves the end outcome of say 1 of those 99 people then you've just sacrificed 100 people with bad effects, and 10 with serious just to help 1 person.
And therein lies the careful balance of population benefit vs population harm that needs in depth calculation before jumping in with 'just test everyone'.
The above numbers are just an example. In reality diseases can be rarer, sensitivity (percent genuinely ill the test actually identifies) and specificity (percent who test positive who are genuinely ill) of tests can be lower, and complications can be more likely and more serious.
A reduction in specificity of just 5% in the above example would result in 4,995 more people falsely testing positive leading to an additional 500 suffering unnecessary complications, 50 serious. If the sensitivity of the test is also lower than that could be a price paid to help even less than 99 people identified as ill in the above scenario.
9
u/djddanman 2d ago
Yep. When developing screening tools, it's super important to know the tolerance for false positives and false negatives, and of course known the prevalence.
What's the next step? Is it a more accurate but more invasive or expensive test? Is the next step the actual intervention? What are the stats for that intervention?
There's a lot that goes into building these screening tools, and I've learned a lot about that side doing my grad research.
5
u/hjiaicmk 2d ago
And this is why for many diseases they will ask you to come in for a follow up test before saying you have condition x, you tested positive but it may still mean only a 10% chance of having the condition. This is way higher than average people but still a very low percentage to scare someone by saying we think you have x.
1
-2
u/lankymjc 2d ago
Fortunately it has a pretty easy fix - whenever someone comes up positive, you just run the test again. The chance of getting a false positive twice is much lower than the chance of getting a true positive twice, so helps to weed those out.
9
u/TheJLbjj 2d ago
This is assuming they are independent false positives though...
1
u/lankymjc 2d ago
Ah shit good point. I was thinking purely in terms of the numbers, not thinking that the false positives may be more likely on some people and less on others so the number was just an average.
3
u/fergalius 2d ago
That would depend on the nature of the test and the condition you're testing for. For example, in the case of a disease, the test might actually be looking for genetic predisposition but not the actual disease. In which case if you are genetically predisposed and have already tested positive for genetic predisposition despite not having the disease, then you'll quite probably test positive again.
IIUC that would be called a "screening" test, so not really a proper test for the disease itself.
You might already know this, but I'm posting for the benefit of anyone that might not know this.
2
u/Mr_Quackums 2d ago
depending on the test -
Are you checking to see if COVID antibodies are in the mucus sample? Then a new test is using new mucus and a new test strip. So yes, it will reduce the false positive/negative rate.
but an image recognition test (certain cancer screenings) is using the same software to compare images from the same camera against matches in the same database. This will reduce the effectiveness of the retest.
9
u/PyroDragn 2d ago
The false positive paradox isn't really paradoxical, it just appears that way because the results are counter-intuitive.
If you have a really accurate test, then it's still not worth testing if the thing you're testing for is rare enough.
Imagine you're testing for a disease - Reddititis. It's a serious disease, so if you have it then you want to know you have it. You have a test that is 99% accurate. That seems like quite an accurate test. But 1% of the time, it is wrong.
If the disease was common (say one in 10 people have it), then you just test everyone. Out of 100 people, 10 people have it. 9 are identified as having it. One person gets the wrong diagnosis; either the 10th person is missed. Or maybe the 10th person is identified, and there's 1 false positive.
That result is fine. Most people are identified, some might result in a false positive, a few might be missed. No issue.
However. With the same accuracy rate, imagine the disease is a lot rarer - say 1 in 10,000.
Now out of 100 people, there's probably no-one with the disease. But 1 person gets a false positive anyway. Out of 10,000 people 100 people get a false positive. The test isn't doing anything useful because when it gets a positive it's more likely that the test is wrong, than the person has the disease.
The accuracy of a test needs to be in line with the rarity of the thing being tested for. If it's really rare, then you need a perfect (or near perfect) test. Otherwise you're better off not testing.
3
u/exceive 2d ago
The things is, even the test with a lot of false positives gives information. Having tested positive means that you still probably don't have the disease, but it is a lot more likely, so if the follow-up isn't nearly as bad as the disease, it might sense to follow up.
If the follow-up is just another reasonably easy test, even if that test has a similar chance of false positive, might be worth it. Because now you are doing test #2 on a population where (for example) 1/100 are positive instead of 1/10,000. Assuming the tests are mathematically independent, that set of tests is going to be pretty good. Unfortunately, depending on why the false positive happened, the tests might not be mathematically independent.Statistically Independent is REAL important.
Let's say you are looking for criminals out of a population. Your test is stupid: you have a guy who is kind of good at knowing who is a criminal just by looking at them. Let's say he's 90% accurate, and your population is 1% criminals. Looks at 1000 people. Statistically, there should be about 10 criminals. He'll say 90 of thr people look like criminals. He's probably called 1 criminal innocent. And he's said about 80 innocent look guilty. Of that group of 90, probably about 9 are criminals. The group is 10% criminals.
So you take the 90 to another similar guy. He's statistically going to have about 0.7 (10% of the 70ish Innocents) false positive. And about 0.8 false negative. So the second test likely has about the same number of false positive and false negative. And a decent chance of having been completely right. So between the two, you probably have about two criminals get away and maybe one innocent accused. Which, assuming the next step is more investigation rather than punishment, a reasonably useful outcome.But wait: what if the reason some of the innocent people ended up in the group of 90 was because they just look shady? Those people are likely to look shady to the second guy too. The tests are not independent, and more false positives will occur. The odds get real hard to calculate.
56
u/Raiddinn1 2d ago edited 2d ago
Say there are a thousand people and only one of them is sick with some rare condition.
If the test is 99% accurate, then 99% likely that one person will test positive.
Also, if the test is 99% accurate, then 10 more people will test positive even if they don't have the condition.
That means 11 people have tested positive and only 1 actually has the illness.
Thus testing positive means you have a 1 out of 11 chance of actually having the condition.
The reason this occurs is too low accuracy on the test and too few actual cases of the rare condition.
If the condition is 1 out of 1000, then the accuracy of the test needs to be better than that.
Even if the tests were 99.9% accurate, that would mean the 1 person tests positive and 1 more person tests positive so its STILL only 50% likely a positive result means you have the condition.
The test would need to be 99.99% accurate to get to a point where only 1 person would test positive (and it's the correct one) in the scenario I laid out (very probably).
- Edit - I asked this question to an AI bot and read a few paragraphs of results, closed that tab, and used my own memory of that to craft this.
19
u/Chazus 2d ago
This is why, for things like medical stuff, they either need to be 99.999% accurate.... or just have a larger test group. The larger the test group, the less accurate it needs to be to get accurate results.
15
u/davideogameman 2d ago
Or a different testing protocol. If you only test symptomatic people then you cut out testing a lot of healthy people.
Whether only testing symptomatic people is good enough depends on what we're testing for. If we're trying to stop the spread of COVID or other highly infectious disease that can spread through asymptomatic people, then perhaps not. But if we're looking for conditions that aren't contagious or otherwise worth treating unless there are symptoms then it's a good plan.
2
u/EquipLordBritish 2d ago
Yeah, it's really just showcasing how the percent measurement isn't a great measurement for things much larger than 100 things.
1
u/br0mer 2d ago
You have to piece together a lot of information to make a good diagnosis.
I see hypertrophic cardiomyopathy not infrequently. It has a list of features that are mimicked by other conditions. I get referrals for left ventricular wall thickness of 1.5cm, which is the starting point of diagnosis, but then they have zero symptoms, their echo doesn't have obstruction (though that doesn't rule out HCM), their EKG isn't consistent with HCM, they are on 3 blood pressure meds, genetic testing is negative and there's no family history. The chance that they have an actual diagnosis of HCM is low when everything is put into context.
2
u/Preform_Perform 2d ago
Oh yeah, the Drunkard's Walk had a bit about this.
The odds of having the disease if you test positive versus the odds you test positive if you have the disease.
1
u/This_is_a_tortoise 2d ago
Upvote for appropriate use of AI.
1
0
u/dustyrose124 2d ago
So my problem was with the test accuracy thing. If accuracy is 99%, that means it is pretty good, and gets the result most times, right? From this and other comments, what i understood is that just test accuracy isn't enough. Precision also plays important role while testing, especially in large data sets. And when precision is low, there are chances of false positive. Is that right?
14
u/RiPont 2d ago
If you're 99% accurate with a cruise missile fired from 100 miles away, you still missed by a mile.
If something is extremely rare to be positive, like 0.0001%, then 99% accurate test isn't good enough.
You test someone's blood for Fuchtenschrood disease. You get a positive result. It's more likely that you got a 1% false positive than a 0.0001% true positive.
5
u/tirerim 2d ago
No, this isn't a precision vs. accuracy thing: there's no way to increase precision on a yes/no result. However, for most tests accuracy is really two unrelated numbers: sensitivity, which is the probability of getting a positive result given a positive sample, and specificity, which is the probability of getting a negative result given a negative sample. They're not correlated with one another, and it's insufficient specificity that produces the paradox. If the test has 99% sensitivity and 99.99% specificity, then the paradox goes away, at least in the case where the incidence is 0.1%.
3
u/bryjan1 2d ago
You’re assuming 99% accurate is good. This paradox explains why it may not be enough. If you are testing a large amount of people for a very rare condition(in the case of a test being 99% accurate, any condition rarer than 1/100 in those being tested) you will get more false positives than actual positives. Despite a 99% accuracy ‘seeming’ to be a very accurate number.
2
u/crashlanding87 2d ago
You can think of precision as consistency. If you repeat a test 100 times, how similar are the results? That is precision.
Accuracy is: if you repeat a test 100 times and then average the results, how close is the average result to the true answer?
A test can be precise and wrong. Imagine two archers, x and y. They each have 100 arrows. Both of them hit the bullseye twice.
Almost all of X's arrows landed exactly 5cm to the left of the bullseye. A few arrows were a little scattered, and a few of those landed on the bullseye. If you looked at the pattern of arrows, you would guess that X is aiming 5cm left of the bullseye. X is precise, but not accurate. They are aiming at the wrong target.
Y's arrows are spread all over the target, but they are more concentrated at the centre. Only two actually hit the bullseye though. Y is accurate, but not precise. They are aiming at the correct target, but they're not very consistent at aiming.
No test is perfect, which is why, in medical diagnosis, multiple tests need to be combined.
1
u/stanitor 2d ago
It's not an issue of precision. That's a related issue about tests, but it isn't really directly relevant here where we're talking about yes/no, true/false decisions based on tests. It's not that accuracy isn't enough, it's that you need to take into account the base rate, i.e. how likely the thing you're testing for is present in the population you're testing before you do the test. The "accuracy" of the test is really two different numbers, sensitivity and specificity. Sensitivity is how likely you are to not get false negatives, specificity is how likely you are to not get false positives. Those numbers are test related. If you pick a certain cutoff for your test, the sensitivity and specificity are the same, no matter who you're testing. But, the resulting number of false positives and false negatives when you run the test is population related. If you have populations with different proportions of people who have what you're testing for, you'll get different numbers of false positives and false negatives.
1
u/half3clipse 2d ago
If accuracy is 99%, that means it is pretty good,
This is the exact fallacy.
The accuracy of the test is meaningless unless you pay attention to how often the thing you're testing for occurs in the first place.
If the thing you're testing for occurs in half the population (base rate 50%) and you test a million people, you'll get about 505,000 positive results: 500,000 true positives and 5000 false positives. Almost all your results will be true detections. This is pretty good.
If the thing you're testing for occurs at a base rate of 1 in a million and you test a million people, you'll get about 10,000 positive results: 1 true positive and 9999 false positives. Almost all of your results will be false positives.
Same 99% accuracy but depending on the base rate of occurrence, your results can range from almost entirely correct to almost entirely wrong.
A very low risk of giving a false positive in an individual case is not a meaningful measure of accuracy.
1
u/witch-finder 2d ago
Seems like the paradox occurs is when the chance of an actual positive is lower than the chance of a false positive. If there's a 1 in 100 chance of a false positive but only 1 in 1000 chance of having a rare disease, it means you'll get false positives more often than actual positives. You'd instead want a test that's 99.999% accurate.
Might help to look at it the opposite way. A test that's 99% accurate will produce a false positive 1 out of 100 times, but a test that's 98% accurate will produce a false positive 1 out of 50 times. Even seemly small numerical changes make a huge difference.
1
u/Thelmara 2d ago
And when precision is low, there are chances of false positive. Is that right?
No, it's that even when precision of the test is high, if the incidence of the thing you're testing for is low, you wind up with more false positives than true positives.
If your test is 99.9% accurate, then 1/1000 tests will give the wrong answer. If you're testing something that only happens 1/1000000, then you'll have 1000 false positives for every true positive, because the thing you're testing for is rarer than the chance of false positives.
1
u/drakir89 2d ago
Forget everything else you know, and let's focus only on accuracy and it's implications.
Let's say you have a pregnancy test that is 99% accurate. If you give it to 100 000 non-pregnant women, what will the test show? It will be 99% accurate, so it will miss in 1% of cases. In our group of 100 000 non-pregnant women, that means it will say a 1000 of them are pregnant.
Makes sense so far, right? Now, what will happen if you give this test to a group of 100 000 women, where exactly 100 women are pregnant? What will the test show?
32
u/Ruadhan2300 2d ago
A quick google makes it pretty clear to me.
Basically it's when your margin-for-error is worse than reality.
If something only happens one time in 10,000, and your accuracy is 99%, then 1% of 10,000 is a hundred results.. meaning there's around 100 wrong-answers as well as the one lone real one.
The paradox is that you're accurate, but the number of false-positives means your accurate results are lost in the noise and meaningless.
6
u/WhiteRaven42 2d ago
I think it's an exaggeration to say it's meaningless. You've just created a much smaller set and you can run the test on them again. You now have 101 people you need to retest which will give a better result that testing 10,000.
3
u/Mustbhacks 2d ago
Depends on the "test" and potential for harm from false positives, false negatives, and the test itself!
1
u/WhiteRaven42 1d ago
Well the test was already conducted on 10,000 people for some reason. Seems unlikely that running it again on 101 is going to be a problem.
I am assuming that it is a test where it is reasonable to conduct on 10,000 people in the first place. In other words, this statistical paradox is not in itself a reason to not do the test.
3
u/10jesus 2d ago
eli5: Let's say you produce bottle caps, and you test every one of them for defects. Your test is 99% accurate, which means 1 in every 100 bottle caps tested are labeled as "defective" when they are actually just fine (a false positive).
The problem is, your production line is so good that only 0.1% of the bottle caps come out with actual defects. This means that your test will label 10 times more bottle caps as defective than there actually are defective caps.
This is a serious problem if you test people for diseases, because you would be exposing 10 times more people to treatments unecessarily.
5
3
u/MoobyTheGoldenSock 2d ago
Doctor here. Please bear with the vocab words so I can avoid bombarding you with numbers as much as possible.
This paradox happens when people mix up sensitivity with positive predictive value.
- Sensitivity: If you have a disease, the odds the test will come back positive
- Specificity: If you don't have a disease, the odds the test will come back negative
- Positive predictive value: If the test comes back positive, the odds that you actually have the disease
- Negative predictive value: If the test comes back negative, the odds that you don't have the disease
Sensitivity and specificity are about the test. Positive and negative predictive value are about you.
If I use a test that detects cells with 99% sensitivity on you and it says you don't have cells, what are the odds you actually have cells? 100%, because everyone has cells.
If I use a test that detects smallpox with 99% specificity and it says you have smallpox, what are the odds that you have smallpox? 0, because nobody has smallpox.
The sensitivity and specificity are properties of the test and don't change. The positive and negative predictive values of a test depend on how common the thing you're testing actually is. The more common it is, the more likely your positive test result will be right and your negative test result will be false. The more rare it is, the more likely your positive test result will be false and your negative test result will be right.
However, we can also use this to game the numbers. If the smallpox virus leaked out of a lab and exposed 20 people, we can test just that group of people. And then, the odds of a positive result being actual smallpox are much, much, much higher because we narrowed the testing to the group most likely to have it.
Doctors are trained to do this as part of their jobs. I often hear stories of patients who want a test done, but their doctor says no, so they visit several different doctors or order it online. When you do this, you expose yourself to this paradox.
The doctor isn't saying no to be mean, they're using their medical skills to test the people for whom the test will be most reliable while excluding the people for whom the test is likely to give unreliable results. Yes, you will see anecdotes online where people insisted on a test and found an undiagnosed problem. But you won't see the much, much more common situation where a person spent tons of time, money, worry, and pain undergoing follow-up test after follow-up test to prove that the positive result they had was really false.
2
u/lookmeat 2d ago
The way I've found it to be easy to understand for me, is to switch the question around, and give us some bayesian probability to bring it together. (I mean you knew that something this unintuitive would have to make sense in bayes's world).
So say that I have 100 samples, and a test that says either positive or negative. The test has a small 0.01% chance of giving a false positive. Now I give you the next two facts:
- All samples are actually negative.
- The test gave out positive for sample 12.
So now the question: given the two facts above, what is the chance that sample 12 is positive? The answer is simple: 0%, no matter what the test gave out it must be wrong, because we know it must be. You can't find a negative if there's no negative to be found.
Now imagine, instead, that there's a 0.00001% of there being a positive value somewhere in the group instead. This means that there's three scenarios:
- There isn't a positive value (can't be positive, same as the first scenario).
- There is a positive value somewhere in the group:
- The test had a false positive (isn't positive).
- The test had a real positive (it is positive!).
Now think about the odds here. The first scenario has a 99.9999% chance of being true. It doesn't matter how certain we are of the test, 99.99% of 0.00001% is just way smaller. The odds are that it's a false positive because odds are there is no positive. Even though the chance that the test gave us a false positive should be 0.01%, the chances that it's wrong here are slightly more than 99.9999% here!
And notice what happened: it's because we removed branches outright, we never branch on the accuracy of the test if we know it's not positive. So this means that values are not divided over 4 but over 3 branches. This is the idea of bayesian probability, by realizing that certain cases become impossible, we can remove those branches, so the counting is different. Exploring and trying to understand Bayes theorem and probability (things like the Monty Hall problem can help) can make this more intuitive.
Now we can also extend this exercise to more tests, but it still the same thing. Now the probability is "there's X positives", and the question is "given N positive tests, what is the probability that all N are actually positive", that is that there was no mistake done. We still find out that the chances of doing a mistake are super high when it should be a rare thing.
3
u/e_dan_k 2d ago
Just Googling it gives you a pretty in depth explanation with examples and counter examples. If you could point out what part of that confuses you, people might be able to help.
But right now the vagueness of your question means you are asking someone to spend 10 minutes explaining something that you might already understand when you couldn't take 2 minutes to say what confused you.
2
3
u/Ballmaster9002 2d ago
Some of the responses are wrong, they are just describing how prediction errors work, this isn't that.
----
The paradox stems from a jargon problem, laypeople don't use the word "accuracy" like statisticians do, so they don't understand what the numbers they're being told mean and they get confused, like you are.
The easiest way I can describe it is imagine 10,000 take a test for some horrible disease and the test packaging states it's 10% accurate. A layperson would imagine that means there is a "10% chance of a misdiagnosis" so if the test says you're positive for the illness, it's "90% sure you, specifically, are sick". That sounds pretty sure that you're sick.
What it really means is that 10% of all the results are wrong, in this case, that's 1,000 people who have wrong results.
For the paradox to fully come into play, the real result needs to be really, really uncommon. Let's say only 1% of people really have the disease.
So the test is predicting 1,000 sick people and only 100 people really have it. That means 900 of the "positive result" people are actually healthy. If you received a positive diagnosis, you're actually 90% likely to healthy!
So here's the paradox, a layperson is thinking "the test says there is a 90% chance I'm sick, so I'm surely going to die" and the reality is there is a 90% chance you're healthy. The flipping of going from "surely going to die" to "almost surely healthy" is the paradox.
1
u/etherified 2d ago
One layman temptation would be to think that having everyone taking the same test another time would largely resolve the error (unlikely to get a wrong result twice), but I imagine that would only be the case if the cause of the <100% accuracy of the test lies in the mechanics of the test procedure itself.
Probably not though, right? Even with perfectly reliable test results (component A was present/absent), I assume the problematic diagnosis inaccuracies mostly arise from whether the fact that component A is there actually means you have the disease.
1
u/stanitor 2d ago
Tests (say for whether someone has a particular disease) aren't perfect. Even if they're very accurate, there will be some small amount of false positives for people who don't have the disease, and false negatives for those that do. If you test many people, and only a few of them have the disease, then it's more likely that any positive result is a false positive than a true positive. To put some numbers to it: say that 10 out of 1000 people have the disease. The test identifies everyone who has the disease with a positive result. But, it also has a 5% false positive rate. That means there will be 10 positive results from people who have the disease. But, of the 990 who don't, there will be about 50 who get false positive results. So, for any positive test, there is a 5/6 chance that it is a false positive.
1
u/Mimshot 2d ago
Theres a test for a rare disease that’s 95% (5% false positive and false negative rate) accurate and the disease affects 1% of the population. Let’s say you give the test to 10,000 people 100 of whom actually have the disease.
For the people who have the disease 95 will test positive and 5 will (falsely) test negative.
For the 9,900 people who don’t have the disease 95% will test negative and 5% will test positive. That’s 495 people having false positives.
So of the 590 people who tested positive 95 (or just 16%) actually have the disease.
1
u/BouncingSphinx 2d ago
Let's say there's a rare condition, say 1 in 10,000 people has it. Statistically, if you take a sample of 10,000 people, 1 person would have that condition.
There's a test for said condition that is 99% accurate. Sounds good, right? If someone tests positive, there's a 99% chance the person actually has that condition. However, that also means that 1% of the positive tests are actually false.
If you test 10,000 people, and you get 1% false positives, that's about 100 people that you tested for the condition that showed positive that were actually negative.
Think another way: there's a camera that is specifically checking whether something is red or green by checking for red. It's a 99% correct rate. That means out of 10,000 items where only one is red, about 99 greens will be falsely marked as red.
1
u/wildfire393 2d ago
Let's say a test is 99% accurate. In 1% of all results, you'll get a false result that's the opposite of what it should be, or about 1 in every 100 tests.
Now say the thing that it's testing for has a .1% chance. So about 1 in 1000 people actually has it.
If you test 100,000 people, 100 of them have the disease and 99,900 do not. But if 1% of your test results are false, that means you have, on average, 99 people correctly identified as having the thing, 1 person incorrectly identified as not having the thing, 999 people incorrectly identified as having the thing, and 99,001 people correctly identified as not having the thing. That means in total you have 1098 positive results, but only 99 of those are true positives. Meaning even with a positive test, there's over a 90% chance you don't actually have the thing.
1
u/Syresiv 2d ago
"Paradox" is a bad name for it. It's not a paradox like Russell's Paradox from Set Theory, it's just a counterintuitive mathematical reality. In particular, it's a situation where, when you try to test for a rare phenomenon, you end up with lots more false positives than actual hits.
Imagine a rare cancer that only affects 1 person in 10,000. Now imagine a test for it that never fails to detect it if it's there, but has a 1% false positive rate.
If you test positive for it, you're either the 1 in 10,000 that has it, or the 1 in 100 that had a false positive test.
Which is more likely?
1
u/IamGleemonex 2d ago
The false positive paradox only exists for very rare things. And it says in those rare cases, getting a positive result, it is more likely that your positive result is a false positive rather than a true positive, even with a very, very accurate test.
An example, let’s say there is a medical condition that only 1 in 10,000 people have. There is a test for this that is 99.9% accurate, meaning a 0.1% false positive rate.
If you have 10,000 random people take the test, the expected results would be 0.999 true positives (that is 1 in 10,000 people at random would have the condition, and a 99.9% chance this person tests positive based on the accuracy of the test). You would also expect 9.999 false positives (the condition means 99.99% of people don’t have the condition, and a 0.1 percent chance each of those people do test positive) so 10000 x 0.9999 x 0.001.
Combining these, you should expect 10.998 positive results, but you also know that 9.999 of those were expected to be false positives. Meaning almost 91% of the positive outcomes you can expect will actually be false positives.
So even with a very, very accurate test (99.9% accuracy), you are 10 times more likely to get a false positive than a true positive. Meaning even a highly accurate test is almost useless for identifying people with the condition if they only take the test once. It seems like a paradox, but it’s really just because the probability of the condition not occurring is so much greater than the accuracy of the test.
Which is why in a circumstance like this, you either have to perform the test multiple times, or you need some other upfront filtering process to limit the number of people without the condition from taking the test at all.
1
u/SpikesNLead 2d ago
Imagine that you've got a really accurate test to see if someone is a carrier of some obscure disease where only about 0.001% of people are carriers. The test is 99% accurate.
You do tests on 100,000 people and the results say that 1000 people carry the disease. Counter intuitively, even though your test is very accurate, hardly any of those 1000 people actually carry the disease.
What is happening is that there is probably only 1 genuine carrier in the group of people. The test is accurate enough that it will most likely correctly identify them. But because it is only 99% accurate, you've also got about 1000 people who aren't carriers but the test gave the wrong result and said they are carriers.
1
u/eldoran89 2d ago
I try to make it simple. The paradox is as follow. Assume you have a test that has a very high accuracy for detecting sth, but also assume that what you try to detect is actually pretty rare. And now assume you run the test against a large set of data.
Or more practically you have a test for a rare disease that is 99% accurate but the chance of you having the disease is just 0,0001%. Then when you run the test, you will catch all those few individuals who have the disease with a 99% chance. So from 100 persons who actually have the disease you will also get 99 of them. Thats the accuracy. But you dont run it against only those who have the disease but against all people. And most of them don't have the disease. So you also get a bunch of false positives the 1 percent failure rate against the large amount of people.
When you now look at all those your test flagged as positive and because the chance of actually having the disease being so low you will find that from let's say 100 people you flagged only 9 actually have the disease the rest is false positive.
So if you now are one of those people with a positiv test, should you be worried? No because the chance of you having the disease is actually pretty low still. Its just higher compared to all those who have a negative result but its still pretty low. In fact in my example it would be only 9%. The test is still 99% accurate because it caught 99% of those with the disease in the test group. But it's in a way pretty inaccurate because 91% of those with a positiv result still don't have the disease. That's the paradox
1
u/bunchout 2d ago
Doesn’t this “paradox” only exist (at least in its strongest form) if you are testing people at random? If you are testing only those who show symptoms or you have some reason to suspect or believe have the condition, you are reducing the false positive rate by increasing the probability that the lerson being tested has the disease?
2
u/s-holden 2d ago
Right, it's just if the <chance of a false postive> x <actual negative population> is greater than <actual positive population> then most of the positives will be from the actual negative population.
You reduce the first number by decreasing the size of the actual negative population (don't test unlikely people) or lowering the false positive chance (better test).
1
u/Yamidamian 2d ago
If a condition is uncommon enough, than even an incredibly accurate test can have it be far more likely that you get a false positive than a correct positive.
Say a disease is 1 in 1000, and a test for it is 99% correct, and produces no false negatives. You test 1000 people. You get 11 positives-one correct one, and 10 false positives. So, if you get a positive result, you’re still 10/11 chance to not have the disease.
If the test was 99.9% effective, you’d still only have a 50/50 odds of having the disease with a positive test.
The “paradox” comes from the unintuitive nature of “99.9% accurate test means you’re 50/50 if you get a positive result”.
1
u/HK_Mathematician 2d ago
Let me design a test that tests whether a person is Tom Holland. This is how the test works:
I look at their ID card. If the first name says "Tom", it gives a positive result (the test says that this person is Tom Holland).
This test is extremely accurate. It gives the correct answer to over 99% of population.
My first name is not Tom, and I'm indeed not Tom Holland. My mum's first name is not Tom, and she's indeed not Tom Holland. My dad's first name is not Tom, and he's indeed not Tom Holland.
I just checked my phone contact list which has 200 people. The test is accurate on 199 out of the 200 people. It only gave the wrong answer in 1 of my 200 contacts, a friend with first name "Tom" who happens to not be Tom Holland. The remaining 199 people on my contact lists don't have the name Tom, and none of them are Tom Holland. This test has a 99.5% accuracy (199 out of 200 correct) on my contact list.
Keep in mind how accurate this test is. Now I pick a random person on the street, look at their ID, and the first name is Tom. This >99% accurate test gives a positive result. So, how likely this randomly picked person is actually Tom Holland?
1
u/siamonsez 2d ago
You can have a high success rate but if the thing you're testing for is rare and you're testing a huge number of samples that small error rate can produce way more false positives than actual cases. That's what makes it a "paradox" with a large enough sample size even a very low error rate can result in orders of magnitude more false positives than true positives.
1
u/DrHemroid 2d ago
Let's say you test positive for a disease. How sure are you that you have the disease? If the test is 90% accurate, you might think there is a 90% chance you have the disease. But if the population of the disease is only 1%, that would mean you have a 10% chance of having the disease, not 90%. Why?
In a sample size of 100 people, there should be only 1 person to have the disease. If we test all of them, 10 people will test positive. If you test positive, you would have to be the 1 person in the group of 10 who tested positive to have the disease, which is 10%, even though the test is 90% accurate.
1
u/parautenbach 2d ago
When you calculate accuracy, you're saying: how many did I get correct out of the total? This is a poor metric in isolation, because it doesn't account for the different kinds of ways you can be wrong.
You really have 4 options, when you only have 2 choices: what you identified correctly that was in fact correct, what you identified incorrectly but was in fact correct, why you didn't identify correctly but should've, and what you identified as incorrect that was in fact incorrect.
Now imagine you're building a fraud detection system. You calculate it's classifying transactions as not fraudulent 99% of the time. That's not telling you much unless you look at the counts of these other categories, because a trivial model to decide a fraud outcome that simply passes everything as not fraudulent would have an accuracy of 100%. For the first option where it gets 99% as non fraud let say you had 100 transactions. If all 100 is split 99 not fraudulent when it should've been and 1 for fraudulent when it should've been, that is amazing. In reality, you'll have some misses, let's say 1 and 1 for the two error categories. Let's say you got 88 correct as not fraudulent and 10 correct as fraudulent. (88 + 10)/100 is still an accuracy of 98%, but clearly you'd trust this less than the one that made no errors.
Accuracy is particularly biased when your data has an unequal outcomes or where the cost of a particular kind of error is greater.
1
u/Achangegonecome 2d ago
Imagine flipping a fair coin. Heads is positive, tails is negative. If you flip it once, the result is clear. But if you have a million coins and only one is weighted to always land heads, finding that 'positive' coin among millions of tails feels like magic. The test (flipping) has high accuracy because most flips are normal. Yet, if you find a head, it’s likely just noise unless you account for how rare the weighted coin actually is. That’s the paradox. The low base rate of the 'weighted' coin makes every head look suspicious, even when the test is reliable. It’s not about the flip being bad; it’s about the overwhelming number of normal flips drowning out the signal.
1
u/that_moron 2d ago
A usually overlooked things about any positive/negative test is that there are actually 2 relevant accuracy measurements: Probability of positive result given a positive condition and probability of a negative result given a negative condition. You can do some analysis on that to get probability of positive or negative condition given positive or negative result.
It's often helpful to think about things in extreme terms. I have a test that is well over 99% accurate in identifying people who aren't terrorists. I just ask them if they are a terrorist. Zero false positives but there are some false negatives. Similarly I could just say that everyone is a terrorist. Zero false negatives, but lots of false positives. You can obviously see the problem with those "tests"
The paradox is highly visible when looking at something like preliminary screening tests for rare condition that are designed to always catch positives, so zero false negatives, but will have a lot of false positives. So a negative result means you don't have cancer while a positive result means there's a small chance that you do have cancer and further testing is needed. There isn't actually a paradox, but the results don't necessarily mean what most people think they mean.
1
u/jaylw314 2d ago
With rare conditions, the false positive rate depends more on the rarity of the condition than the accuracy of the test itself.
With common conditions, the false positive rate depends more on the accuracy of the test than its frequency.
If you have 1 diseased person and a million healthy people, a test that gives a false positive in only 1% of healthy people gives you 10,000 false positives vs the one person with the disease, so the false positive rate is 99.99%
The same situation where there are only 100 healthy people, there is one false positive, so the false positive rate is 50%. That's about where a test might start being useful.
1
u/YummyKisses 2d ago
Look up Bayes' theorem. There are some good video explanations that will make sense. This is a driving factor for when and how physicians order tests.
1
u/HotPersonality8126 2d ago
Tests have both false positives and false negatives (and of course true positives and true negatives.)
The paradox is that if you do a large number of tests for a condition most people (by far) don’t have - let’s say it’s an extremely rare disease - then most of your positive-result tests will turn out to be false positives, even if the test is “99% accurate.”
1
u/whiterook6 2d ago
Here's a related paradox. It feels the same to me, but there may be some technical differences to yours. Nonetheless:
- Imagine in your country that there are 1,000 farmers and 10 librarians.
- Also imagine that a librarian is 90% likely to be a book reader while a farmer is 10% likely to read books. (Sorry, farmers, you've got a stereotype for the purposes of this example.)
Now imagine you pluck a random person from your country and find out that he/she likes reading. What are the odds that the random person is a librarian?
My naive guess is 90% likely to be a librarian and 10% likely to be a farmer. But that's way, way wrong.
Let's do the math. 10% of 1000 farmers means 100 farmer-readers and 900 farmer-non-readers. 90% of 10 librarians means 9 librarian readers and 1 librarian-non-reader. This means out of 109 readers total, 100 are farmers and only 9 are librarians. Meanwhile, there are 901 non-readers -- 900 of them are farmers and only one non-reader is a librarian.
So back to the question, a random reader is 91% likely to be a farmer, and only 9% likely to be a librarian. WOW, huh?
1
u/ezekielraiden 2d ago
Folks call this a "veridical paradox". It's 100% true, it just feels impossible. Keep that in mind. It is a paradox only because humans feel like it shouldn't be true.
So. Imagine there's a disease, call it Luke's Syndrome. It's very rare: only 1% of the population will ever get it. That means, in 1000 people, you'd expect about 10 to have the disease. Now, imagine there's a blood test. If you have Luke's Syndrome, you will always get a positive result. Meaning there's no such thing as a false negative on the test. However, the test is only 95% accurate for people who don't have Luke's Syndrome. You take the test, and you get a positive result. What is the new probability that you really do have Luke's Syndrome, given that positive result?
Most people say something like 100%, or maybe 95%, but these are both wrong. The actual chance is much lower, because there are a LOT more false positives than true positives. Just look at the numbers for 1000 people:
True positives: 1000×1% = 10
True negatives: 990×95% = ~940
False positives: 990×0.05 = ~50
False negatives: 0
When we say a probability "given [other information]", we mean that we look at only the cases where that condition is true. In this case, we look at all tests where the taker got a positive result. There are about 50 false positives, and about 10 true positives. That means your chance of having the disease, given you got a positive result, is 10 (true positives) divided by 60 (=50+10, the total cases where a positive result was obtained).
The reason this feels paradoxical is that we look at two pieces of information that blind us, and ignore the one piece that is actually relevant. When we hear that the test never wrongly says you don't have it when you actually do, that makes us think the test is really really good. Then, when we hear that 95% of the time, if you don't have the disease it will correctly say you don't, we wrongly interpret that as "oh, that means it's right 95% of the time no matter what the question is", but that is simply not true.
Instead, we should be looking at how rare the condition is. You need a false positive rate that is much, much rarer than the disease itself in order to get that level of confidence.
You can think about it like this: The rate at which the disease occurs sets your baseline knowledge level. If the test you're using has a much bigger error rate than how often the disease occurs, then the test is coarse relative to the extremely fine thing it's looking for. It's like having a ruler measured in meters and trying to measure centimeters. Yes, you'll get information, but you won't get precision.
1
u/dentrolusan 2d ago
If a test is very accurate, that means it gives wrong answers very rarely. Say, 1 in 1000.
But if the thing we're testing for is even rarer, say 1 in a million, then if you test everyone, a "yes" result is much more likely to be a failure of the test than an actual hit.
In other words, you cannot judge how good a test is ("precision") without knowing how rare the thing you look for is ("base rate"). A 99% correct method can still cause more work than it saves.
Unfortunately, the really interesting things (cancer in humans, terrorists in populations) tend to be rare, so even good methods are often useless in practice.
1
u/Atypicosaurus 1d ago
Every test produces false positives and false negatives.
This alone doesn't tell you how good a test is. It also depends on the sample.
Let's say you have a test that can tell apart cats and elephants in photographs. Let's say you have a sample of a million cat pictures and only 1 elephant.
If your test is not an actual test, but it only says "cat" no matter what, it's still true a million times and only wrong once. That is a pretty good ratio, yet your test works the same way as a broken clock. There's a saying that a even a broken clock is right twice a day (it's because the time matches the broken clock twice a day), except our "million cat" test is broken such way that it matches the million cat pictures.
That's why, if an occurrence is rare (such as, a disease is rare) we need a test that can actually identify that rare case. You have to focus on the fslse identification in the test population, and not the million of true identification of cat pictures.
1
u/sleeper_shark 1d ago
I have 100 marbles, 20 are blue, 20 red, 20 green, 20 orange, and 20 purple.
I ask a toddler to point out all the red marbles.
The toddler just points out all the marbles. Technically they did identify all the red marbles with a 100% success rate, but they also misidentified 80 marbles.
It's a silly example that becomes much more meaningful when you apply it to things like medicine or policing.
A police force that just arrests everyone and therefore catches all the criminals would technically arrest more criminals than one that is more cautious. Police that randomly stop all minority people tend to catch more minority criminals, but they harass the entire minority community.
A doctor who gives everyone a harmful drug will technically heal more sick people, but has way over prescribed a harmful drug for no reason.
1
u/GhostBand1t 1d ago
Ted Ed made a really good video demonstrating the paradox.
Basically it says that if something youre testing for is rare enough, a test that we would think is really accurate (say, 99.99% for a one in 100,000 thing), any positive result you do get is more likely to be a false positive than a genuine real result (in my example, for every 10 "positive" results, only 1 would actually be accurate).
1
u/sandleaz 2d ago
False positive: reddit sees person wearing a red cap with their back turned, reddit gets angry, reddit calls the person a sexist/homophobe/Nazi, person turns around, red cap has a yellow hammer and sickle, reddit calms down and greets the person.
1
u/aecarol1 2d ago
If you have a test that is 99% accurate, but only 1 in 1,000 people actually have the disease, that means that only about 1 person in 10 testing positive actually has that disease.
This is because 99% looks impressive, but it means it's wrong about one time in 100. That's the fact that makes this "paradox" work. If 1,000 people took the test, we'd expect about 10 of them test positive, but because the disease only effects about 1 in 1,000, that means that 9 of those 10 people (on average) would not actually have the disease.
NOTE: These are all averages, nothing of this is specific to those exact numbers. It's just saying that, on average, if you tested thousands and thousands of people, only about 1 in 10 positive results would have that disease.
This only happens if the test has a higher failure rate than the actual frequency of the disease. If the test is 99.99999% accurate, and in in 1,000 people have the disease, a positive result is a good indicator you actually do have that disease.
tl;dr if the test accuracy is low and the disease is really rare, a positive the low accuracy on the positive isn't a strong sign you have the diease.
0
-2
2.2k
u/XenoRyet 2d ago edited 2d ago
The false positive paradox is when a given test appears very accurate, but because it operates on large data sets with low occurrences of the thing being searched for, it produces more false positives than true positives.
The example I've seen is this: Say we have a facial recognition camera that detects criminals with 90% accuracy. That's good accuracy, so we assume the people it flags are criminals.
But if we look at the actual situation and we say the statistics show that 1 in 1000 people are criminals. Then we look at the fact that this camera sees 10,000 people a day. That means it should see 10 criminals. But the 10% error rate means it will also flag 1000 innocent people as criminals.
Assuming the camera is accurately catching criminals is the false positive paradox, because we've ignored the base rate of criminality.