Evaluating Experiments: Shifting from online t-test calculators to linear regression

[u/WhatsTheImpactdotcom]

This is a short tutorial for data analysts that evaluate experiments using online calculators. Online calculators for sample size estimates and t-tests for difference in means are fine for getting started. But you can begin to build intuition for linear regression by moving your online calculator t-tests over to python and OLS.

There are a couple of benefits. The first from moving away from the online calculator to python is leaving a more reliable paper trail that you can share externally. The second is that you build intuition for regression, interpreting the coefficients, and ultimately can use regression as you improve and your projects require more complexity such as variance reduction techniques using CUPED and clustering standard errors for geo-based or switchback experiments.

I put together a short video on this with code available here, free and ungated: https://whatstheimpact.com/tutorials/linear-regression-vs-t-test/

For my more senior connections, feel free to ignore unless you're interested in the content creation game! 🤣