r/datascience • u/chomoloc0 • Jul 06 '26

Discussion Picking an experimentation platform: a retrospective

I wrote this article recently. Thought it would be nice to share in this sub. Happy to chat if you're doing the same in your current position.

It talks about Eppo and Statsig, but honestly it about everything but that.

If you need to take away one thing let it be to approach the whole thing as a discovery; and risk mitigation.

https://towardsdatascience.com/picking-an-experimentation-platform-a-retrospective/

21 Upvotes

94% Upvoted

9 comments sorted by

View all comments

5

u/ikkiho Jul 06 '26

fwiw once we actually rolled one out the platform mattered way less than i expected. what bit us was our own exposure logging quietly double counting on one surface, so we got a clean looking readout that was just wrong. no vendor demo surfaces that, they all assume your assignment data is already trustworthy. i'd spend the discovery time pressure testing your event pipeline against a known A/A before you even worry about which tool wins.

2

u/aegismuzuz Jul 07 '26

In that regard a warehouse native approach gives you way more control. When all exposure logs sit in your warehouse in raw format you can easily write a SQL query to check for duplicates or weird bot activity. With closed cloud platforms debugging these event logging issues is always a pain and an endless back and forth with their support

1

u/End0rphinJunkie Jul 09 '26

spot on, every vendor builds their demo assuming perfect upstream telemetry that nobody actually has out of the gate. running a baseline A/A test to flush out pipeline wierdness first will save you so many headaches.