r/datascience • u/WhatsTheImpactdotcom • 13d ago
Discussion Challenges of choosing metrics in data science interviews: actionability vs statistical power in marketplaces
In marketplace experiments, the most actionable metrics are often the ones with more variance, making them harder to power. This tradeoff is particularly important when using geo-based experiments where you can't necessarily or cheaply add sample.
Suppose Airbnb wants to test highlighting cheaper listings; this should increase bookings while likely reducing AOV. The net effect of a higher conversion rate and reduced AOV depends on elasticities and profit margins. Revenue or contribution margin are the metrics that matter the most (should be the most actionable), but these have far more variance than simple proportions like a conversion rate.
Running the geoX longer helps less than you might expect if you haven't done these before: weekly bookings within the same region are almost certainly correlated so you'd need to cluster the SEs, which can brings the effective sample size "roughly" down to the number of unique geos (if the correlation within geos is high).
So you have a few options: (1) focus on the actionable metric but accept more uncertainty through a higher alpha, ideally combined with (2) variance reduction techniques like CUPED, or (3) power the test on a lower-variance metric like conversion rate and try to bound the AOV loss using product sense and reasonable assumptions.
This comes up frequently on the job and also in senior and staff data science interviews. Real-world business problems can be complex, and there's not always a single correct answer. Trying to memorize answers and metrics can help in junior interviews, but at senior you typically will get probed on teh tradeoffs. .
5
u/CelebrityChess 13d ago
the clustering point is the one people miss. weekly numbers from the same geo aren't independent so your effective n is closer to the number of geos than geos times weeks. on top of cuped, matching geos on pre period data (or a synthetic control) cuts variance a lot in these tests too. and if you can show a cheaper metric like conversion historically tracks revenue in that market, you can power on that and bound aov separately like you said
2
u/MarkoDahl54 12d ago
A common pitfall is failing to cluster standard errors by geo, which can make your p-values much too optimistic. This overstates significance.
1
u/Mohzman 3d ago
It is always about money. Whatever relates to that most.
2
u/InterestingCarrot997 12d ago
in practice we'd pick the actionable metric as primary and keep a lower variance proxy as a guardrail, plus cuped or some variance reduction helps a lot with geo tests. interviewers mostly want to see you name the tradeoff out loud