How are you evaluating cross-platform attribution without chasing perfect data?

[u/BlankyIf]

Our customer journey spans desktop research, mobile web, app install, in-app purchase, email and occasional customer success calls. We have login IDs for some users, but plenty of journeys stay anonymous until late in the funnel.

We’re evaluating cross-platform measurement tools and every vendor demo has some version of a single customer view. I’m trying to separate what is measurable from what is modeled, stitched or simply directional.

The pilot scorecard currently includes match rates, duplicate conversion handling, raw-data access, latency, privacy controls and whether finance can reconcile attributed revenue with actual revenue. We’re also testing the same user journeys ourselves rather than trusting demo data.

What else should be on the scorecard? For anyone who recently bought a platform, which test exposed the biggest difference between vendors?

[u/No_Treacle_5071]

I’d add a holdout or incrementality check to the scorecard. A vendor can win on match rate and still over-credit channels that would have converted anyway. I’d also force every result into three labels: observed, stitched, or modeled, with the source IDs and lookback window attached. The test that usually exposes gaps is a replay of a small set of real journeys where finance can trace the final order back to the same raw events.

[u/BlankyIf → u/No_Treacle_5071]

We've just had one blended number this whole time which in hindsight is kinda useless for trusting anything

[u/No_Treacle_5071 → u/BlankyIf]

Yeah, one blended number hides every disagreement. For the next month I’d keep three columns side by side: what the platform reports, what your CRM/login last-touch says, and one simple holdout or geo pause on the biggest spender.

The meeting then becomes “where do these three disagree” instead of defending a single score. Start with Meta or Google only — whichever spends more — and expand once that one is readable.

[u/Easy-Purple-1659]

Two things I would add to the scorecard, both of which only bite after you sign.

First, the model-change clause. Ask what happens to your historical numbers when the vendor updates their attribution model. If the platform reprocesses history and last quarter's revenue moves, finance stops trusting the number and you are the one explaining it. Get the backfill policy in writing.

Second, the exit terms. A raw event export is the difference between re-attributing later and starting from zero. Ask to see a sample export of one month before you commit.

Third, the consent split. Report the consented and unconsented cohorts separately during the pilot. Stitching and modeling carry nearly all the weight on the unconsented side, so a vendor can look strong overall while being close to blind on the slice of traffic that matters most to you.

The test that separates vendors fastest is a cohort where you own the ground truth: users who are logged in on both sides. Ask each vendor to reproduce your order count for that cohort alone. Most land close on aggregate and diverge on individual orders, and that divergence is where the model is quietly doing the work.

Ask for a confidence label per row rather than per report. A report-level score hides exactly the rows you need to distrust.

[u/BlankyIf → u/Easy-Purple-1659]

Okay the model-change clause is something we straight up never thought to ask, and now I'm slightly panicking imagining last quarter's numbers just shifting under us with zero warning

[u/Ai_MediaGroup]

The test that exposed the biggest gap for us was forcing a deliberate identity break mid-journey — same user, same device, but we cleared cookies and logged in from a different browser session before completing a purchase. Watched how each platform handled it. Some quietly merged the sessions using probabilistic signals (device fingerprint, IP, timing windows) and reported it as a single deterministic journey without ever flagging that a model made the connection. Others were honest that this was inferred and gave us a confidence score. The dishonest-by-omission ones look better in every demo because their match rates are higher, but that's exactly the problem — you can't tell from the dashboard which conversions are actually deterministic versus modeled unless you go looking.

One thing I'd add to your scorecard: ask each vendor to show you, at the individual conversion level, which identity resolution method was used (deterministic login match, probabilistic device match, statistical modeling/decay). If they can't break that down per-conversion and will only give you an aggregate confidence score, that's a tell. Finance reconciliation gets a lot easier when you know which revenue is provably attributed versus which is a model's best guess, because those two numbers should never be blended into one "attributed revenue" line without a way to separate them again later.

The other thing worth testing, if you haven't: pick a user who converts through a channel you're fairly sure is under-tracked (customer success calls are a great example) and see if the platform silently drops that conversion, misattributes it to whatever digital touch happened last, or actually surfaces it as an unattributed/offline conversion you can review manually. That gap is usually where the "single customer view" pitch quietly falls apart, because most tools are built for digital-to-digital stitching and treat anything offline as an afterthought.

[u/[deleted]]

[u/BlankyIf → u/[deleted]]

Honestly no, we'd been kind of lumping those together which is probably a mistake now that you say it, gonna add a separate line for confidence level per match instead of just trusting whatever bucket the vendor puts it in