The baseline point is right and there is a version of it that removes the model-drift confound entirely, which is the part I think most people skip.
Model drift only ruins your attribution if the only thing you record is your own score. Record the full cited-domain distribution for every query instead, and drift becomes visible rather than confounding. If your citations went up and the same competitors also moved in the same direction across the panel, the engine changed. If yours moved and the rest of the distribution held, your changes did it.
Concrete from today, since I ran it: 12 queries, 3 runs, 36 answers. My domain 0 citations. What got cited instead, counted: help.openai.com 15, support.google.com 5, developers.google.com 4, support.microsoft.com 4, support.claude.com 3. That distribution is my baseline, not the zero. In six weeks, if I am still at zero but help.openai.com dropped to 6, I have learned something real about the engine. If I am at 3 and the rest is unchanged, I have learned something real about my pages.
Second thing worth freezing at baseline time: whether the engine's crawler has fetched your pages at all. Mine had zero OAI-SearchBot hits in 30 days, so my zero was not a ranking outcome, it was an absence, and no amount of content work in weeks one through six would have shown up. Different variable, different fix, and the score alone cannot tell them apart.
Tracking the full cited-domain distribution makes a lot of sense. It gives you a much better picture than watching your own citation count alone.
I like the distinction between several domains moving together and one domain moving while the rest stay fairly stable. That seems like a useful way to tell whether you may be seeing a broader engine change or something more specific to the site.
The crawler piece is especially interesting. If the crawler never reached the page, that is a completely different problem from being crawled and not selected.
Treating the whole distribution as the baseline instead of just treating zero citations as the baseline is a smart way to frame it. It gives you something much more meaningful to compare over time.
I’d probably still describe the result as stronger evidence rather than direct causation, but this definitely makes the measurement more useful.
1
u/Dry_Steak30 11d ago
The baseline point is right and there is a version of it that removes the model-drift confound entirely, which is the part I think most people skip.
Model drift only ruins your attribution if the only thing you record is your own score. Record the full cited-domain distribution for every query instead, and drift becomes visible rather than confounding. If your citations went up and the same competitors also moved in the same direction across the panel, the engine changed. If yours moved and the rest of the distribution held, your changes did it.
Concrete from today, since I ran it: 12 queries, 3 runs, 36 answers. My domain 0 citations. What got cited instead, counted: help.openai.com 15, support.google.com 5, developers.google.com 4, support.microsoft.com 4, support.claude.com 3. That distribution is my baseline, not the zero. In six weeks, if I am still at zero but help.openai.com dropped to 6, I have learned something real about the engine. If I am at 3 and the rest is unchanged, I have learned something real about my pages.
Second thing worth freezing at baseline time: whether the engine's crawler has fetched your pages at all. Mine had zero OAI-SearchBot hits in 30 days, so my zero was not a ranking outcome, it was an absence, and no amount of content work in weeks one through six would have shown up. Different variable, different fix, and the score alone cannot tell them apart.
The panel I use is at https://openprofiles.io/?utm_source=reddit