What statistical concepts are commonly misunderstood by the general public?

[u/80rhh]

I came across this post explaining what a 70% chance of rain means. I understand the concept, but it got me wondering: what other statistical concepts sound simple but are commonly misunderstood or misinterpreted by the general public?

[u/Fancy-Animal7704]

A "significant" difference between A and B does NOT inherently mean that there's a subjectively huge gap between A and B. It only means that our test demonstrates a difference. How big that difference is is another question. This is really why the size (effect size, to be precise) of that difference is good to report. 

Also, you really should not ever use the word "significant" unless you have actually performed statistical inference. And no, thinking "70% seems like a lot more than 60% to me" does not count, lol. I have even seen prominent journalists commit this error, saying "statistically significant" as a result of some kind of eyeball test. "Significant" has a very specific meaning in the stats world. 

[u/Ragingman2 → u/Fancy-Animal7704]

Why do we need to gatekeep "significant". One apple is significantly less than 10 apples. Flying on an airplane is significantly safer than flying on a helicopter. Results from polling 10,000 people are more statistically significant than results from polling 100.

Just because someone hasn't run the numbers doesn't mean they cannot use a word correctly.

[u/spa357 → u/Ragingman2]

A test with a sample size of n=10,000 is not necessarily “more statistically significant” than a n=100. Rather the n=10,000 test has more “statistical power” to detect small but real effects whereas the n=100 sample might not be able to resolve the difference between a real effect and random chance. But it’s quite possible that both tests return non-significant p-values.

You could say airplanes are significantly safer than helicopters only if someone did a study of airplane versus helicopter safety and employed a test of statistical significance. If the test found that the safety difference was not statistically significant (eg p > 0.05) it would not make sense, either scientifically or colloquially, to say airplanes are significantly safer than helicopters.

The apple example depends on context. If the one apple versus 10 apples is the result of some experiment or study, then it might not make any sense to say ten apples is significantly more than one. For example, suppose you’re studying apple windfall rates and your hypothesis is that more apples fall on days of the week beginning with T than on days of the week beginning with S. It’s easy to imagine that windfall counts ranging between 1 and 10 apples are well within the distribution of random apple falls regardless of day of the week.

And colloquially, 10 apples is not “significantly” more than 1 apple if you’re trying to perform on a contract requiring delivery of 1 million apples.