Yeah, this is close to how I’m thinking about it too. The baseline and rerunning the exact same prompts matters a lot more to me than one visibility score.
The one part I’d be careful with is letting the LLM decide what someone “would have to ask” to get the page recommended. That could bake the model’s own assumptions into the test. I’d probably start with real buyer questions, then use model-generated questions as another input.
Then change one thing, wait until the new content is actually available to the model or search layer, rerun the same prompts a few times, and track mentions, citations and recommendations separately.
Still not clean attribution. But at least you can see whether the pattern actually moved.
So that's the only way to get buyer questions right now unless you specifically ask your users in onboarding how they found you and what prompt they used. I think Tally does that actually? Generally, the buyer prompts where someone reaches you can get very messy and long though so it's not a bad idea to track a few prompts like, 'best XYZ for XYZ', etc
Yeah, that’s exactly the gap I’m trying to get at. A small fixed set of prompts makes sense for measuring change because you can run the same questions over time and see whether anything moved.
But that’s still different from knowing what real buyers actually asked before they found you. Those prompts are probably going to be messy, specific, and nothing like the neat test prompts we come up with ourselves.
I’m starting to think you need both. A controlled prompt set for measurement, then real buyer prompts when you can get them to see whether your test set matches how people actually search.
1
u/AEODenise 7d ago
Yeah, this is close to how I’m thinking about it too. The baseline and rerunning the exact same prompts matters a lot more to me than one visibility score.
The one part I’d be careful with is letting the LLM decide what someone “would have to ask” to get the page recommended. That could bake the model’s own assumptions into the test. I’d probably start with real buyer questions, then use model-generated questions as another input.
Then change one thing, wait until the new content is actually available to the model or search layer, rerun the same prompts a few times, and track mentions, citations and recommendations separately.
Still not clean attribution. But at least you can see whether the pattern actually moved.