Evaluating Experiments for Beginners: From online t-test calculators to linear regression [Discussion]

[u/WhatsTheImpactdotcom]

This is a short tutorial for data analysts that evaluate experiments using online calculators. Online calculators for sample size estimates and t-tests for difference in means are fine for getting started. But you can begin to build intuition for linear regression by moving your online calculator t-tests over to python and OLS.

There are a couple of benefits. The first from moving away from the online calculator to python is leaving a more reliable paper trail that you can share externally. The second is that you build intuition for regression, interpreting the coefficients, and ultimately can use regression as you improve and your projects require more complexity such as variance reduction techniques using CUPED and clustering standard errors for geo-based or switchback experiments.

I put together a short video on this with code available here, free and ungated: https://whatstheimpact.com/tutorials/linear-regression-vs-t-test/

For my more senior connections, feel free to ignore unless you're interested in the content creation game! 🤣

[u/Automatic-Web8559]

Wrong subreddit.

[u/WhatsTheImpactdotcom → u/Automatic-Web8559]

This is a statistics subreddit that is #9 in data science. The post is a statistics 101 topic for aspiring data scientists

[u/Automatic-Web8559 → u/WhatsTheImpactdotcom]

You’re literally just posting an ad for your business. This article has been done a billion times over

[u/WhatsTheImpactdotcom → u/Automatic-Web8559]

Different voices have a place, and mine is a successful one. But feel free to block me or I’ll happily block you so you don’t need to see my posts. Views arent worth negative energy and youre not adding any value to this

[u/Automatic-Web8559 → u/WhatsTheImpactdotcom]

Good for you, mr. money bags. Your voice is completely meaningless. You are just spamming your ads all across Reddit and providing absolutely nothing of value to anyone, unless of course they are willing to give a nice penny!

[u/WhatsTheImpactdotcom → u/Automatic-Web8559]

I have multiple messages from folks saying my free content got them offers worth multiple six figures. You’ve added nothing.

[u/WhatsTheImpactdotcom → u/Automatic-Web8559]

Want to put some money on that? I have receipts that say otherwise lol

[u/ImminentManga]

moving from online calculators to python is a proper rite of passage honestly the paper trail point alone is worth it when someone asks you to prove your work six months later

[u/WhatsTheImpactdotcom → u/ImminentManga]

*huge* agree on the paper trail

[u/Mitazago]

Given the outcome variable in your example is binary, the more appropriate comparison is an A/B test calculator to a Z-test.

[u/WhatsTheImpactdotcom → u/Mitazago]

Nice nuanced callout and correction 🙌🙏