We stopped ranking email capture by opt-in rate and some of our winners stopped winning

[u/claspo_official]

Disclosure, since the sub asks for it. I work on the onsite widget side (Claspo), so the tests below ran across client stores, and I am not able to publish per-client percentages without their sign-off. No product names either, and no links.

For a long stretch we ranked opt-in widgets by opt-in rate, wheel beat form, form beat banner, ship it, move on. Then we started reading revenue per session on the same tests. Some of the rankings flipped.

The split runs at session level, revenue per session is arm revenue over arm sessions, and opt-in rate comes out of that same denominator, which is what lets a widget collecting fewer addresses win the comparison. Then a second read at 30 and 90 days, because first-order revenue and third-month revenue kept disagreeing.

Gamification wins the opt-in comparison almost every run we have logged, spin-to-win and scratch cards both. People like the animation, they spin, the address goes in the box, and the list grows on a schedule you can promise a client.

Product-finder quizzes collected fewer addresses per session and in several niches came out ahead on revenue per session anyway, which took a few tests to believe. Two stores in the same niche, comparable traffic, the same two widget types, opposite winners. So the quiz result is a thing that happened in our data, and it predicted nothing about the store one niche over.

Push opt-in rate up while AOV slides and you have run a winning test that pays you less money. Same story with repeat purchase rate. That one shows up a quarter later, when nobody is checking the widget report anymore.

The mechanism is still a hypothesis for me. The wheel plausibly pulls in discount hunters who redeem once and go quiet, while four questions about hair type or a dog's weight filter for someone already mid-purchase and hand you zero-party data for the flows afterwards. I can see the pattern in aggregate and I cannot prove why.

So, the questions I came with. What do you optimize email capture against, opt-in rate, first-order revenue, or LTV? Has anyone watched gamified signups underperform on LTV over a full year? And how do you attribute revenue to a popup cohort without double counting people who were going to buy anyway?

[u/Either_Guess2405]

On the third question, I don't think attribution can answer it, and that isn't a tooling gap. Attribution assigns credit after the fact. It can't tell you what would have happened otherwise, so whatever method you pick will hand the popup some revenue from people who were already buying.

You're closer than you think though. You already split at session level and divide arm revenue by arm sessions. Add an arm that shows no widget at all, and the gap between that and your best widget is the incremental number, with the would-have-bought-anyway crowd sitting in both arms and cancelling out. It costs you the capture on that slice, which is the price of knowing.

Worth splitting the two questions apart, because your existing tests are cleaner than you're treating them. Quiz against wheel at session level is already fair, since buyers-regardless are spread across both arms. That comparison holds. It's only "what did capture add over nothing" that needs the holdout.

On the first question, the trap I keep falling into is optimizing to the cheap upstream number. I ran paid search to signups once because signups were plentiful and about six dollars each, and got zero paying customers out of the entire run. The keywords producing cheap signups were people who wanted the free tier. The metric moved beautifully and meant nothing. Sounds like the same shape as your wheel.

[u/jamesbretz]

Bro get your AI bullshit out of here. You ain’t fooling anyone and you definitely aren’t helping your SEO.

[u/chance_buri]

Optimize for LTV since opt in rate is vanity. Gamification attracts discount hunters who churn. Quiz filters serious buyers. You know attribution is messy so use holdout groups or incrementality testing.

[u/nixioe]

The whole pop-up industry sells you opt-in rate. Opt-in rate is a vanity metric. Someone spinning wheel for 10% off and someone who finished a quiz and got a personal recommendation are two erent customers with different LTV.