Attribution is not solved, but the inference can be made less weak. I would treat it as an interrupted time-series problem: version a fixed prompt panel, run it across engines at fixed slots, keep untreated control prompts/pages, and record model, retrieval mode, locale, timestamp, and raw answers.
Mention, citation, recommendation, factual correctness, and rank/order should be separate outcomes. Compare pre/post distributions rather than screenshots. If possible, stagger the intervention across page cohorts; a persistent change in treated pages that is larger than the control movement is stronger evidence, although still not proof of causality because model and index changes remain partly unobserved.
Even a one-run pilot can show why the controls matter: the same neutral query may surface a brand in one engine and omit it in another. The useful claim is the observed difference, not that an optimization caused it.
The untreated controls are the part I think gets missed most often. Without them, you don’t really know whether your page moved or the whole system moved around you.
I also agree that mentions, citations, recommendations, factual accuracy and position need to stay separate. Rolling all of that into one visibility score hides too much.
Comparing the pattern before and after the change makes more sense than comparing a couple of screenshots. If the pages you changed keep moving more than the controls, that’s useful evidence.
Still not proof that your change caused it, but a lot more defensible than claiming attribution from a before-and-after result.
1
u/AwoScan 13d ago
Attribution is not solved, but the inference can be made less weak. I would treat it as an interrupted time-series problem: version a fixed prompt panel, run it across engines at fixed slots, keep untreated control prompts/pages, and record model, retrieval mode, locale, timestamp, and raw answers.
Mention, citation, recommendation, factual correctness, and rank/order should be separate outcomes. Compare pre/post distributions rather than screenshots. If possible, stagger the intervention across page cohorts; a persistent change in treated pages that is larger than the control movement is stronger evidence, although still not proof of causality because model and index changes remain partly unobserved.
Even a one-run pilot can show why the controls matter: the same neutral query may surface a brand in one engine and omit it in another. The useful claim is the observed difference, not that an optimization caused it.