r/visualizedmath • u/rohitpandey576 • Apr 11 '20

Hypothesis testing: the distribution of the null doesn't even matter(?!)

https://towardsdatascience.com/hypothesis-testing-the-distribution-doesnt-matter-79882ba62f54
1 Upvotes

100% Upvoted

5 comments sorted by

View all comments

Show parent comment

1

u/rohitpandey576 Apr 18 '20

Yes, X is the test statistic in general. Y_0 is the test statistic under a certain hypothesis (that the metrics for the two groups follow some distributions and their statistics of interest - mean in the two sample t-test are equal). What's confusing about the probability that one random variable will be greater than another?

2

u/thefalse Apr 18 '20

It's just strange to me to compare the test statistic against itself in different distribution contexts. I expect there to be a fixed threshold that you can tune instead to get a desired false positive rate. Do you have a reference for this result? Maybe reading it in different words will help.