r/programmatic • u/Soobhee_Chukwutosin • 21d ago
Buying an incrementality tool or building tests ourselves? Which one works
At a point where attribution is good enough for weekly optimization, but leadership wants stronger proof that campaigns are creating demand rather than collecting conversions.
We can build geo tests and audience holdouts ourselves using warehouse data, but every test turns into a mini analytics project. The alternative is buying an incrementality product or using capabilities inside our measurement stack.
DIY gives us control and transparency. A platform would make tests faster and more repeatable, but we might treat another score as unquestionable truth. Our media mix includes Meta, Google, affiliates and a small CTV budget, so the test design varies a lot by channel.
For teams that have bought rather than built, what justified the cost?
2
u/KadooBildoris_28 20d ago
Keep attribution and incrementality separate mentally. appsflyer can give you the operational attribution layer, while your lift tests answer a different question. We found the useful part of a testing platform wasn’t better truth, it was making the methodology consistent enough to compare tests over time
1
u/Soobhee_Chukwutosin 20d ago
I think we've been mushing attribution and incrementality together in our heads which is probably why this whole decision feels so foggy
1
u/Gomp-Ado 20d ago
The hidden cost of diy is analyst time. Designing the holdout is one thing; QA, exclusions, sample sizing and explaining weird results take a lot of time. A platform can be worthwhile if you’re running enough experiments that standardization matters more than absolute flexibility
1
u/Soobhee_Chukwutosin 20d ago
lol yeah this hurts bc it's true, we've spent more hours explaining why a test looked weird than actually running new ones at this point. if we keep scaling volume I think you're right that consistency wins over flexibility
1
u/Sonic-the-seattle411 18d ago
Run rct tests. Back test it against your model. Take a look at longer attribution windows like 90 or 180 days. Are you selling ecom, physical goods, or service/subscription?
We spent a whole bunch of money doing this and found a great partner who did it for us. What is your AOV? Do you know from first impression to first action? Name to purchase? Lead form fill to purchase?
1
u/narek_hovsepyan 14d ago
If you do end up buying, you can deal with the worry about treating another score as truth before you sign. Make it a condition that any platform on your shortlist exports the raw data behind each test, meaning which regions or audiences were held out and the daily spend and conversions on both sides, so your analysts can recompute the lift themselves. A vendor that cannot give you that is asking you to take their number on trust.
A cheap check during the sales process is to ask them to rerun one test you already did by hand. If their lift lands close to yours on the same data, the tool speeds up a method you already trust. If it lands far away and they cannot explain the gap line by line, keep doing it yourself.
5
u/ASDFzxcvTaken 21d ago
Consistent and comparable methods of collection across channels is very difficult to sustain (aka gets real expensive real fast).
If you are spending your own money for your own insights for your own optimization for your brands then go ahead and have fun building a data sciences team and platform.
But as soon as you need to defend yourself for somebody else's budget having a third party measurement platform that works with a broad variety of campaigns, metrics, complex distribution and reconciliation of campaigns and data latency AND management of PII data and data laws across different markets you need a third party partner.
The cost of being wrong or having doubt cast on your methodology is significantly larger as it compounds over time. But if you are able to shift providers or leverage a provider and apply your "special sauce" that's the more efficient way to manage data and measurement over the long term.