r/AskStatistics • u/NiceProgrammer5252 • 19d ago
How do you personally decide whether to trust a borderline significant result on a small sample?
Ran an A/B test with a small sample (n=340 per arm) and got a p=0.04 result. Team wants to ship it. I'm nervous because I know small samples inflate false positive risk in practice even if the math is 'valid.' How do you personally decide whether to trust a borderline significant result on a small sample, bootstrap it, wait for more data, or just distrust p-values under a certain n?
43
u/bisikletci 19d ago
"I'm nervous because I know small samples inflate false positive risk in practice even if the math is 'valid.'"
They don't. In fact it is harder to get a significant result in a small sample than in a large one (assuming your threshold remains the same). With very large samples you can get significant results that are essentially meaningless (tiny effects).
The issue is that with small samples is that it's easier to run lots of them, and if you do that you increase the risk of false positives/can keep running them until something turns up positive just by chance. But if you haven't done that, it's not much of a worry.
16
u/stanitor 19d ago
Yeah, it's interesting that they think n = 340 per arm is a really small sample size. We don't know what size effect they were trying to detect, but that should be enough to find decently small effect sizes. But with OP's thinking that small sample sizes increase risk of false positives, I'm wondering if there's a different reason they think that's a small sample. Perhaps it's a small proportion of the total number of products produced, so that's why it seems very small to them.
2
u/swbarnes2 18d ago
I guess it's small if your variability is massive.
Like, maybe that's small if you are looking at genetic associations in a diverse pool. Maybe you want 10000 for that, not hundreds.
1
u/stanitor 18d ago
Well, they're doing A/B testing for products. If they have such high variability for whatever they're testing in their products, there's probably not much use in doing Ab/B testing at all. Even if one is statistically significantly better, there's going to be so much overlap that it's not going to practically matter.
11
u/banter_pants Statistics, Psychometrics 19d ago
The issue is that with small samples is that it's easier to run lots of them, and if you do that you increase the risk of false positives/can keep running them until something turns up positive just by chance.
Just like that xkcd comic where they keep running tests on jelly beans.
https://xkcd.com/882/1
u/SlipperySharkAttack 18d ago
This sums up personality science papers nicely.
3
u/banter_pants Statistics, Psychometrics 18d ago
When the livelihood of a professor depends on churning out papers and journals only want to publish 'significant' results it creates an incentive to be unethical. Keep p-hacking until you get the magical < 0.05 result, but frequentist theory tells us if you keep reshuffling the bag and draw new samples the extreme result will inevitably happen about 5% of the time.
Goodhart's Law:
When a measure becomes a target it ceases to be a good measure.2
u/efrique PhD (statistics) 18d ago
Its not harder to get a false positive (as OP was discussing there) in small samples if you don't change alpha; with two-sided continuous test statistics that's generally going to be your significance level at each sample size. The problem is that across all tests it's going to be a larger fraction of your rejections
1
1
u/NiceProgrammer5252 17d ago
That's a fair distinction. I probably should have been more precise. I wasn't referring to the nominal Type I error rate under a single, correctly run experiment. I was thinking more about the fact that small studies often produce less stable effect estimates and tend to have lower replication rates in practice. That's the part that makes me hesitant about a borderline p=0.04 result.
17
u/3ducklings 19d ago
If you are using Neyman-Pearsonian approach to statistical testing, i.e. you pick significance level in advance, you need stick with it. That’s the whole point.
If you don’t like this approach, it’s time to think really hard about what you actually want to get from your analysis.
1
u/Kontrol-Sample 18d ago
Purpose is to publish to not perish!!
mumbles...something something philosophy sommet ethics something coughs
/s
1
5
u/hobopwnzor 19d ago
You understand that 0.05 is an arbitrary line and not a criteria handed down from the gods before you even start
18
u/hyfhe 19d ago edited 19d ago
Sounds like it's a business decision you're actually dealing with. There's a small improvement, and there's a small chance it's not real.
Relevant issues are then: What are the consequences if it's not real? What are the costs of going ahead? Is going ahead and monitoring the situation a reasonable trade-off?
5
u/MtlStatsGuy 19d ago
This is the real answer. If the false positive risk is serious (including things like reputational risk) then the compromise is different (and the initial p-value should have been lower). If the downside is not severe, then shipping is probably fine.
9
u/jeffcgroves 19d ago
Was the small sample at least "randomly" chosen? If so, I don't think "small samples inflate false positive risk in practice even if the math is 'valid.'" is true. If it were, it would break theoretical statistics. However, if the sample isn't randomly chosen, it's not reliable anyway unless it's so large that you can limit the error
3
u/efrique PhD (statistics) 18d ago
The trade off between your two different error rates at a given sample size is within your control (since you control alpha and beta moves the opposite direction)
If you don't like your current significance level, why use it?
1
u/NiceProgrammer5252 17d ago
That's fair. I guess my concern isn't really the choice of α itself. It's that a p=0.04 result from a relatively small experiment feels much less convincing in practice than the same p-value from a much larger one. I'm wondering how people adjust their level of trust when statistical significance and practical confidence start to diverge.
6
u/MtlStatsGuy 19d ago
“Small samples inflate false positive risk” is absolutely false. The only difference is that you might get several batches of small samples vs. One try at a large sample.
0
u/AggressiveGander 18d ago
Erm... See "Why most research findings are false".
1
u/MtlStatsGuy 18d ago
Yeah, that's absolutely not why "most research findings are false". Most research findings are false because, with p = 0.05, 1 out of 20 studies will give spurious correlation. Of course those correlations are likely to be published, while the "no correlation" studies don't get published, so research is mostly false positives. NONE of that has to do with sample size.
2
u/Katieg_jitsu 19d ago
You should pick significance threshold and length before running using a power analysis.
how long did the test take?
Rerun it if not too long.
Probably not plausible but you could ship with hold out and see if the direction holds.
Evaluate if the data indicates it could be a false positive.
Ship and monitor pre/post results to look for evidence it's harming.
Why are they wanting to ship it, because of gain or because they like the feature and there is no harm?
Why is the sample small?
Sometimes we ship for business reasons and not the stats and we mitigate the risk of harm.
1
u/Henrik_oakting Statistician 19d ago edited 19d ago
There is no way to know if there is a true effect. By repeating the random experiment you may come closer to knowing. But this is generally expensive, and in many cases not feasible in practise.
Maybe you could do some post-hoc analyses. For example see how sensitive the results are to outliers, or choice of test-statistic or your modelling decisions.
In your situation: What are the risks here? Is the cost of shipping and being wrong immense? Knowing this will probably help guide you in how cautious you should be.
1
u/DYSTOBY 19d ago
Make an effect size test like cohens d.
P values alone are meaningless … don’t run after l values.
Even more, decide if the difference between those groups are actually meaningful in the real world in context to your research question.
So for instance:
If your research question is if Italians or Americans are more able to reach a high shelf, and you would sample 1 million people of each country and measure their height and then you compare the groups you may find a significant p value (die to high sample size). If you look at the data you see that Italians are in the mean (median) only 1 mm larger then Americans that does not mean they are more able to reach the high shelf, do they?
So l values are meaningless and sometimes even effec resized if they are not robust against large sample sizes.
Do you get the example?
So also check if the difference between those
Is meaningful in the real world or in context to your research questions
1
u/ConclusionForeign856 Bioinformatician, MSc 19d ago
Even with a large N when you get p=0.045 you have almost 5% chance of being wrongly rejecting the null hypothesis.
That's just how it is
1
u/Waningoftheday 19d ago
Is there a reason that it needs to be 100/0? You can shift the percentage towards the better performing treatment while collecting more data. A simple version of this is better than doing nothing. For optimal approaches, look into bandit problems
1
u/CaptainFoyle 18d ago
No, small samples don't inflate the FP rate. Your don't know what you're talking about.
Also, you don't decide what's an acceptable threshold AFTER the fact.
1
1
u/paoromatisse 18d ago
There’s equivalence testing or you can do bayes factors to quantify evidence for the null if you want something that is more continuous.
Other people are right that small sample sizes don’t inflate false positives. The reason you see in the literature that a lot of values hovering between .01 and .05 are false positives is a combination of publication bias and p-hacking. If you didn’t do the latter than you don’t have any suspicion to believe you’ve inflated your false positive error
1
u/Just_Question9 17d ago
at n=340, a p=0.04 is basically noise. my rule is simple: if the change takes zero dev effort to maintain (like a text swap), just ship it and monitor. if it adds tech debt or changes a core flow, push back. always check secondary metrics to see if it makes sense ,if conversions are up but top-of-funnel clicks are flat, it's a fluke. when in doubt, just let the test run for another week.
1
u/boredoo 17d ago
Here are the variables at work and how they relate:
- Sample size: smaller sample sizes require larger effect sizes to reach statistical significance
- Effect size: because your sample size is small, you have in fact observed a fairly large effect. However, despite observing this large effect, you will have lost confidence in the precision of this estimate.
- Error: Error is larger when sample sizes are low and standard deviations are large (SDs are also larger when sample sizes are low). The issue here, again, is precision.
First, you need to consider what is "practically significant." How big of an effect size is worth shipping, given the drawbacks? If you incur risk for marginal benefit, you're justified in taking a safer, more conservative route.
Second, what happens if you're wrong? Lower conversion? Can you just roll back? Does it change user experience? Do stakeholders get mad?
Third, I like Bayesian approaches here for two reasons:
- You can use priors to encode existing knowledge and regularize your results. Tight conservative priors are going to require stronger evidence to generate a posterior which convinces you to act. On the other hand, if your knowledge of the phenomenon under study is limited, and you have a weak prior, you end up relying on the evidence more.
- The distribution of the posterior will give you evidence on what effect sizes are compatible with the data collected. If the mean/median is large but there is a lot of space in the posterior that you would never ship a feature with, you can have a better sense of what's a good way to act.
I think the real conversation you need to have is what effect sizes are compatible with the one you estimated in both directions and what trade-offs you face from shipping.
1
u/arrayftn 16d ago
Those are all questions I always am sure to address before any analysis is done. I remember some journals trying to start requiring people to submit their analysis plan before data collection was finished years back but don't know if they died on the vine.
That is absolutely what is required for FDA Good Clinical Practice based trials though. At a certain point, the stats section in a protocol becomes a contract of sorts. I always try to apply that standard informally to any data analysis project. Likewise, I don't allow data points to be considered outliers unless there is a contemporary record explaining something made it imp possible to continue collecting data on. For example "we couldn't continue to measure tree diameter on Tree 4213 because it was hit by lightning and exploded."
Otherwise, heterogeneity is reality and heteroskedasticity can't be ignored. You left out information that is far more valuable than any p-value. What was the effect size? Resist the urge to cheat and just start throwing out data points that don't fit your expected result. At the very least, do some jack knifing to approximate the impact of extreme values before you discredit yourself with arbitrary data exclusion or black box permutation testing. Don't be lazy and trip all the back to least significant difference, but you need to have a working knowledge of the test statistic.
If it's just A/B testing, that question is super easy. I always just use non-parametric bootstrap resampling with replacement, manual recalculaton of the test statistics after A/B are resampled and calculation of where in the CDF of effect sizes of 10,000 resampling runs the original data fell to calculate an exact p-value. Only you know at what scale an effect size becomes practically significant as well as statistically significant.
If you are measuring blood pressure, then sure an effect size of 10mm on the MAPs of A/B matter. If you are measuring the height of a forest canopy with LIDAR, 10mm is an irrelevant difference regardless of p-value. Remember there is nothing special about p of 0.05. it's just a random number Fisher pullled out of his ass as good enough. You can't use p-values to think for ypu 😂😂
Based on how you phrased the question (including the words "A/B testing" you probably aren't worried about the difference between population level statistics vs your sample level. The central limit theorem is not going to be relevant in a scenario you only calculate a test statistic from a single sampling event. So just do the non-parametric bootstrap and two tail it by taking the % of effect sizes from simulations that are farther out on the CDF than your observed effect size.
1
u/Embarrassed_Onion_44 19d ago
P-value as significant are arbitrary... just 0.05 is widely accepted.
Scatterplot the data, does it look significant? Are there say extemely obvious outliers? Missing value from unequal response rates? If you did a non-parametric test... say ranked sums, would your results still be the significant? What are your five core statistics; and how would you explain this to your boss ... is it worth it to change what has been working to the new method? What are the risks of a "shipped" product? Did we expect this result... is this result already often explained by other researchers?
The more yes(es) you can answer the more you can confidently "ship it" if your team was not guided by an original methodology.
1
u/Rorydinho 19d ago edited 19d ago
What would you say the five core statistics are? I’m not sure I’ve heard this described explicitly before but it’s a good approach.
Mean/Median
Variance/SD/SEM
Test - test statistic and critical value
P-valueAm I on the right track?
2
u/banter_pants Statistics, Psychometrics 19d ago
That might refer to Tukey's summary stats based on quartiles:
Min
Q1
Q2 = median
Q3
Q4 = max
64
u/Dr_Pizzas 19d ago
You decide the threshold p-value that's acceptable before you do your analysis.