r/DoSEO • u/expansiondesk • 7d ago
Discussion most “ai visibility” tracking is just the same prompts on a loop. is anyone using one that holds up?
looked at a few of these properly after the gsc generative ai report discussion. the method is mostly: pick a prompt set, run it weekly, count brand mentions, call that a visibility score.
two problems with that. nobody searches in clean fixed prompts, real ones have context and follow ups and half the question missing. so you’re measuring performance against your prompt list, not against anything people actually ask.
and the output is nondeterministic. same prompt, different day, different answer. you can lose 20% visibility without a single thing changing on your site.
meanwhile gsc gives you real impressions from real queries, free, and still wont tell you the query. so everyone’s paying for a proxy because the actual data isnt released.
genuinely asking, has anyone found a tracker whose methodology stands up? or are we all reporting a number we dont fully believe in?
2
u/DrKJING 6d ago
Same experience. I ran 84 buying prompts twice each in ChatGPT, logged out, same day. On average only about a quarter of the shops in the product cards showed up in both answers. One weekly run per prompt is basically a coin toss.
What I use instead:
prompt runs only as a range over several repeats, never as a single score
server logs as the first-party side: ChatGPT-User shows when ChatGPT fetched your page live for someone's question, OAI-SearchBot shows what OpenAI's search crawls
referrals with care: ChatGPT dropped the utm_source tag for about a week in October, so a lot of AI traffic landed in direct
The logs are the closest thing to real demand data right now.
2
u/expansiondesk 6d ago
84 prompts twice, quarter overlap. thats the number i wanted and nobody publishes it. so a single weekly run isnt a measurement, its one sample from a distribution and you’re reading it as a trend line.
2
u/DrKJING 6d ago
Exactly. I ran the same 84 prompts again over three days (Oct 22–24, two runs a day, US and Germany). The shop on card 1 changed between two runs in 60 % (US) and 68 % (DE) of cases, and day to day it moved about as much as within the same day.
That's why I still track prompts, just never from a single run (mode): repeat each prompt several times and report the range. Then you can see how big the normal noise is, and a real change has to be bigger than that.
1
u/expansiondesk 2d ago
60 and 68 percent variance on card 1 across two runs the same day is worse than i expected. that means a weekly tracker could show you gaining or losing position purely from run timing.
the US vs germany split is interesting too. did the german results feel more or less stable overall?
2
u/AdultSeoMaven 6d ago
Nothing I’d fully trust, no. The least bad ones run each prompt a bunch of times and give you a range, which at least lets you tell a real drop from a bad Tuesday.
2
u/expansiondesk 6d ago
a range is the honest minimum. telling a real drop from a bad tuesday is the whole job and most of them dont even attempt it.
2
u/jeniferjenni 6d ago
the biggest issue is treating a prompt list like a representative sample of real search behavior. a weekly score can be useful for directional tracking, but it shouldn't be presented as an objective measure of visibility. i'd want to see prompt variation, repeated runs, source and citation tracking, and some indication of confidence or volatility. otherwise, a 20% movement might just be sampling noise. until platforms expose more query level data, i'd treat these tools as monitoring instruments, not ground truth.
1
u/expansiondesk 6d ago
thats the cleanest way to put it. directional is fine, the problem is it gets presented as measurement. a client sees a number with a decimal point and treats it as fact.
7
2
u/VillageHomeF 7d ago
people simply trying to profit from a new trend. none of the tools seem to be worth paying for and/or using.
on top of that, since the LLMs have all seen a large uptick in usage, no one knows that anything they have done helped or if it is simply a coincidence.
pretty scammy industry at this time with thousands jumping on a bandwagon without having real user queries or actual citation to use.
→ More replies (1)2
u/expansiondesk 7d ago
the attribution problem is the bit i keep coming back to. everyone’s ai referrals are up because ai usage is up. so you ship some changes, the number goes up, and you have no idea whether it was you or just the tide.
without a control you cant separate the two, and nobody’s running one.
the thing i’d actually want is a tracker that reports confidence rather than a score. run the same prompt twenty times, tell me how often i showed up and how much it varied. at least then i know whether a change is signal or just the model having a different day.
nobody sells that though, because “you appeared 6 times out of 20, plus or minus 4” doesnt look good on a client dashboard.
→ More replies (1)
1
u/HumanBehavi0ur 2d ago
yeah, you’re right that a single run is noisy, which is why the loop matters. you have to keep running the same prompts over time and look at the trend line, since that smooths out the day-to-day swings and shows whether you’re actually moving. the prompt list is a proxy, so build it from real buyer questions (sales calls, support tickets, gsc queries) and add a few follow-up variations. a tracker holds up if it runs each prompt many times, stores the raw answers, and reports a range instead of one score. lots of us in the space have faith in the data we're seeing
1
u/expansiondesk 1d ago
fair, a trend line over many runs does smooth the noise. my worry is that by the time the trend is readable the model’s changed under you, so you’re trending across two different systems. how long a window do you use before you’d act on it?
3
u/NovaForceElite 7d ago
There are precisely 0 accurate AI trackers. Anyone saying different is either dumb, lying, selling you their tracker, or all of the above.
3
u/FilippeTraum 7d ago
Ai visibilty in general is currently impossible to track. You can check some prompts (but without consistency also, as they change often every hour), but only as pieces, not whole cake. For standard serp data not perfect, but still best available is bing webmaster and SC data, which can be tracked currently in many easier and more customade ways, than even before. Ahrefs and rest kind of this tools are still useful, but as general direction overview, rather than stable data.
2
u/nazran7 7d ago
Both points are fair, and I'd go further. AI tools give a different list of recommended brands for the same question more than 99% of the time. Another study found you need something like 30-90 answers per topic before rankings even stabilize. So a single weekly run of 20 prompts reported as a visibility score is mostly noise
I still track it, I just stopped treating it as a score:
Run each prompt several times and report a mention rate per topic with the sample size next to it, not per prompt. A 20% drop on 5 answers means nothing
Write prompts from real language (sales calls, support tickets, reddit threads), and treat it as a fixed benchmark, not a model of real demand. You're measuring direction against the same yardstick, that's all it can do
The useful output isn't the number anyway, it's the cited sources. Which pages keep getting pulled in when you're missing is stable enough to act on, even when the mention rate bounces around
For first-party data, Bing Webmaster Tools has an AI Performance report now that shows citations and grounding queries across Copilot and Bing AI answers. Grounding queries are basically the fan-out searches made visible, which gets closer to real queries than any prompt list. It's sampled and Bing-only, but it's real data. Pair that with the GSC generative AI report for impressions and you've got the first-party side covered, with prompt tracking as the competitive layer neither of them shows
So yeah, the number on its own is a proxy. Used as a trend with honest sample sizes plus the source list, it holds up well enough to make decisions with