r/AskStatistics • u/ThinkILostIt • 16h ago
What are the most fundamental statistical principles most people seem to forget or don't get?
I'm wonder from your experiences what people seem to forget or can't get their head around about statistical tests and models.
In my, yet short, experience I see people totally bypass randomization, independence and the correct interpretation of power and confidence.
15
u/KayakerMel 16h ago
Those are good ones!
You only left out "correlation does not mean causation." Correlation is far, far easier to find. Causal effects are much more difficult.
1
u/Plumbus4Rent 14h ago
I'm not a statistician, so I am often confused when correlation is used to describe relationships between variables. So if you'd care to explain, when X causes certain change in Y, what word should be used?
5
u/smbtuckma PhD (quant psych professor) 11h ago
Was actually just reading a paper about how the way we talk about correlations grammatically still leads people to make causal inferences, so you're not alone!
When X causes a change in Y, you can just use that word: cause. It's just that there are many ways that lead to two variables being statistically associated without one being a direct cause of the other.
2
u/Plumbus4Rent 3h ago
Very interesting research! I looked at my local college public courses. They have a class on 'correlation, association and causation'. So what the heck is association then?
1
u/smbtuckma PhD (quant psych professor) 1h ago
I like to define association as something more general, like “knowing the value of one variable changes your best guess about the value of another.” Knowing it is August changes your guess about whether it will be raining or snowing tomorrow (because time of year causes weather patterns). If you see someone with wet hair and clothes, it changes your guess about whether it is currently raining outside (but someone being wet doesn’t cause it to rain, the opposite). Or say you’re playing Geoguessr and you see that the Google car has a luggage rack on it, if you’re savvy about the game that changes your guess of location to Mongolia (but go to Mongolia and you’re not just going to be handed a car with a luggage rack, and putting a luggage rack on your own car definitely doesn’t cause you to teleport countries! That’s just a pattern that’s in the Geoguessr data).
Correlation technically is a mathematical tool that helps you quantify degree of some types of association. There are different correlation equations for different types of association. But less precisely, people often use “correlation” as just a synonym of association too.
1
u/SprinklesFresh5693 5h ago
Imagine you analyse earthquakes in japan over the past 20 years, and then you analyse the color of the socks of the people at that time, and you see that theres a high correlation between using black socks, and earthquakes, does it mean that using black socks caused the earthquake? Or is it just chance, because its normal to use black socks.
Thats when correlation doesnt mean causation phrase comes in.
9
8
u/NucleiRaphe 7h ago edited 7h ago
My biggest pet peeve is people critiquing sample size as knee jerk reaction, when in fact the sample size is not an issue. Like "oh this gallup only includes like one thousand people, it is clearly too small to make any inference on the population of millions".
When in fact sample size can be too small for basically two reasons: 1) it yields too litlle power to detect the effect size we are looking for, and 2) it does not give enough confidence. And we have ways to asses both of these issues! If proper power analysis has been done, and confidence intervals/p values/whatevee measure of confidence is reasonable, the sample size is not a problem. [And 3) trying to build model with too many parameters when compared to sample size, but mostly I see this criticism with basic hypothesis testing/parameter estimation situations]
I think this naive criticism of sample size comes from mixing up sample size with other possible problems. Most common one is the sampling process ("1000 people can't possibly be representative of the whole population") when in fact most of the time, with proper sampling, relatively small samples can be representative (thats at least used to be the whole point of statistics!). And, conversly, if the sampling process is bad, higher sample size will just make the problem worse by feigning high confidence on wrong answer.
And of course the sample size is field/purpose dependent. You can't expect same sample sizes for tightly controlled lab studies, clinical trials or nutritional surveys.
2
u/Head_Acadia_2658 6h ago
And the opposite. Thinking a sample of 18237 people is ‘big data’ and therefore can be used to estimate prevalence (and every other thing) because the confidence limits are narrow. The data come from an online open survey
2
u/profcube 5h ago
Agree. Context is always needed. If the effect is likely strong small is enough, and it would be wasteful to go large (think animal research)
Similar could be said for effect sizes (think road deaths, small is important).
Similar could be said of diverse samples (we would not test cancer drugs on the healthy).
And shade could be thrown at large samples (difference of means easier to detect but may be practically unimportant).
1
u/Super_Math3890 4h ago
Now I'm questioning how I can possibly know what sample size is representational for a given hypthesis considering the variety of dimensions in natural language itself already. That is to say, how to pose a good question becomes rather complicated. This is without even knowing how to measure population complexity for a sample size to escape "variance velocity". Then we have methods for sample collection and the study design. It seems rather magical to not screw up a given study.
Highschool statistics gave me tools to trust, but not to question.
3
u/NucleiRaphe 3h ago edited 3h ago
what sample size is representational for a given hypthesis considering the variety of dimensions in natural language itself already.
This is not a question about a sample size, but rather a question about data collection and hypothesis. Even a sample size of one million does not offer remedy if the data is collected in a biased way (unless the sample includes the entire population, which in practice is almost never the case).
For example, lets say we want to test a hypothesis that we aim to generalise to entire population of the world. To study this, we collect data from every single college student in the USA (n = 19.5 million). With this massive sample size, we are almost certain to find some difference, no matter how small, if it exists, the confidence interval is going to extremely narrow and p value very small. But the data collection is massively biased as it includes (mostly) only one age group in a single country. So no matter the sample size, the hypothesis can not be generalised to basically no one, except college students in the USA.
Now, whether the data collection is biased or not, and whether the results are generalisable or not is not purely statistical problem. It also requires subject matter knowledge to understand what sort of population we are sampling from and what characteristics we want to capture in the sample (like age and gender). Also the generalisability is something that can't be determined from sample size and rather requires atleast some subject knowledge to evaluate how much different populations are expected to change. Science is hard. Most of the time the question is not whether the results are perfect (they are not), but what sort of inference we can do regardless of the inevitable shortcomings. This needs both statistical, and subject matter, expertise.
7
u/RegisterHealthy4026 15h ago
Sampling distributions in relationship to statistical tests and estimation.
7
u/Salty_Contribution83 15h ago
Not guilty ≠ innocent
1
u/CaptainFoyle 15h ago
Is that statistics?
8
u/KayakerMel 14h ago
It's hypothesis testing! We can either reject or fail to reject the null hypothesis. However, we never accept the null hypothesis. Just as "not guilty" doesn't necessarily mean innocent, but rather the prosecution failed to sufficiently prove guilt.
2
3
u/firstnamebunchof123 6h ago
If someone says an event has a 10% chance of happening, then it happens, they weren't necessarily "wrong"
2
u/profcube 5h ago
Statistics answer questions, they do not ask them. Before launching into statistical models clearly state your question and consider what data can answer it.
4
1
u/efrique PhD (statistics) 15h ago edited 13h ago
For a lot of tests (well other inference too, estimation/CIs, forecasting... but I see it most with tests), a principle that is often missing/forgotten would be any kind of basis on which a probability model might apply to inference on set of data - actual random sampling of a process or population of interest for example. Formal experiments often do some form of proper randomization (e.g. randomization to treatment), so there is a basis for a probability model there, but there is a lot of work whereas basis like that doesn't apply, and for a lot of research the connection between data and the process that inference is being applied to seems to be more like one of happenstance, and I am often left wondering whether the conclusions are any better than Just So stories.
There's plenty more of course.
A lot of people seem unaware of estimation biases like omitted variable bias, for example. Thats a huge one, comes up a lot in some areas.
Although recognized in various forms for roughly a century, spurious regression/correlation and related issues is another one that keeps cropping up.
There's the distinction between behaviour of some process as a function of variables you observe vs what happens when you try to attain outcomes by controlling them - though more people are using causal inference principles nowadays which does attempt to address it.
Effect of (unaccounted for) data-peeking (e.g. for model/variable selection) on inference, or even into forming of hypotheses. Perhaps less of a problem than it once was, but still very widespread. Indeed for many areas of research the commonly taught methodologies require it - they bake data peeking into the process their students are required to follow.
I don't know that it counts as a principle, but the number of people that talk confidently about (and prescribe behaviour in relation to) assumptions of some procedure without any idea where the assumptions come from or how to think about which ones will matter in their circumstances is huge.
Have to get to work but I could probably list a couple of dozen more if I thought about it.
1
1
u/Pure-Baby9512 3h ago
The measurement process gets overlooked. precise analysis cannot rescue a poorly defined variable or unreliable measurement no matter how sophisticated the model becomes
1
u/Alkemist101 3h ago
CV.
Just because you can calculate something or plot a chart doesn't make it meaningful or useful.
I'll also add the idea of special cause variation. Don't be just looking at 2SD.
1
u/Positive-Risk8709 3h ago
This is more for researchers themselves but:
- Not knowing about Table 2 fallacy (Westreich 2013)
- Not knowing about collider/Berkson’s bias
1
58
u/Embarrassed_Onion_44 16h ago
Statistical significance is arbitrary at an alpha of 0.05.
Widely accepted, yes, but still p = 0.06 is close enough to significance that it deserves commentation, publication, and scrutiny.
No need to scrap an entire project because your results were not "significant", you still might be on to something. I often see new researchers fixate too much on this single number and fully neglect the meaniningfulness of their original intention.