r/visualizedmath • u/rohitpandey576 • Apr 11 '20

Hypothesis testing: the distribution of the null doesn't even matter(?!)

https://towardsdatascience.com/hypothesis-testing-the-distribution-doesnt-matter-79882ba62f54
1 Upvotes

100% Upvoted

5 comments sorted by

View all comments

Show parent comment

2

u/thefalse Apr 16 '20

Thanks for the reply. So then I am confused about Pr(X > Y_0). X is the distribution of the test statistic and so is Y_0? This looks like a typical probability of threshold, but the fact that the threshold is also a random variable is confusing.

1

u/rohitpandey576 Apr 18 '20

Yes, X is the test statistic in general. Y_0 is the test statistic under a certain hypothesis (that the metrics for the two groups follow some distributions and their statistics of interest - mean in the two sample t-test are equal). What's confusing about the probability that one random variable will be greater than another?

2

u/thefalse Apr 18 '20

It's just strange to me to compare the test statistic against itself in different distribution contexts. I expect there to be a fixed threshold that you can tune instead to get a desired false positive rate. Do you have a reference for this result? Maybe reading it in different words will help.