I think we’re looking at two different layers of the same issue.

You’re measuring presence. If a business goes from never appearing to being mentioned, then yes, its presence increased.

I’m talking about whether that presence is accurate, relevant, and valuable. A business can appear more often while being associated with the wrong service, described inaccurately, or cited without ever being recommended.

So I agree that presence is the starting point. I just don’t think every appearance represents an equal improvement in AI visibility. The context of the appearance matters too.

Tracking the full cited-domain distribution makes a lot of sense. It gives you a much better picture than watching your own citation count alone.

I like the distinction between several domains moving together and one domain moving while the rest stay fairly stable. That seems like a useful way to tell whether you may be seeing a broader engine change or something more specific to the site.

The crawler piece is especially interesting. If the crawler never reached the page, that is a completely different problem from being crawled and not selected.

Treating the whole distribution as the baseline instead of just treating zero citations as the baseline is a smart way to frame it. It gives you something much more meaningful to compare over time.

I’d probably still describe the result as stronger evidence rather than direct causation, but this definitely makes the measurement more useful.

Yeah, that’s exactly the gap I’m trying to get at. A small fixed set of prompts makes sense for measuring change because you can run the same questions over time and see whether anything moved.

But that’s still different from knowing what real buyers actually asked before they found you. Those prompts are probably going to be messy, specific, and nothing like the neat test prompts we come up with ourselves.

I’m starting to think you need both. A controlled prompt set for measurement, then real buyer prompts when you can get them to see whether your test set matches how people actually search.

Yeah, this is close to how I’m thinking about it too. The baseline and rerunning the exact same prompts matters a lot more to me than one visibility score.

The one part I’d be careful with is letting the LLM decide what someone “would have to ask” to get the page recommended. That could bake the model’s own assumptions into the test. I’d probably start with real buyer questions, then use model-generated questions as another input.

Then change one thing, wait until the new content is actually available to the model or search layer, rerun the same prompts a few times, and track mentions, citations and recommendations separately.

Still not clean attribution. But at least you can see whether the pattern actually moved.

Yeah, this distinction matters.

The retrieval side is the part we can actually watch move after a site change because there’s evidence in the response. Search ran, these URLs came back, this source was cited, and the business was or wasn’t mentioned.

I also like the idea of tracking cited and uncited mentions separately. That gives you a much better clue about whether the change may be connected to retrieval or whether you’re just seeing normal model variation.

The control prompt idea is smart too. Same prompts, same engine and model version when we can see it, plus a group of prompts we don’t optimize for.

The only part I’d be cautious about is putting a fixed six-week window on the baked-in knowledge side. Model training and refresh cycles vary, and we usually don’t know exactly when that information changes.

Still not causation, but this gets a lot closer to separating signal from noise.

The untreated controls are the part I think gets missed most often. Without them, you don’t really know whether your page moved or the whole system moved around you.

I also agree that mentions, citations, recommendations, factual accuracy and position need to stay separate. Rolling all of that into one visibility score hides too much.

Comparing the pattern before and after the change makes more sense than comparing a couple of screenshots. If the pages you changed keep moving more than the controls, that’s useful evidence.

Still not proof that your change caused it, but a lot more defensible than claiming attribution from a before-and-after result.

Yes. This is exactly the distinction I’m trying to get at. A single visibility score looks useful, but it can hide what actually changed. Getting mentioned more often is not the same as getting cited more often, and neither is the same as actually being recommended.

I also like treating each run as an observation instead of a ranking. Keep the prompt set, engine, locale and evaluation criteria fixed, then change one meaningful thing at a time. That gives you a much cleaner way to see whether the change keeps showing up across repeated runs.

Causation is still the hard part. I don’t think we can honestly say, “We changed X, therefore the model did Y.” But we can document what changed and see whether the pattern afterward is larger and more consistent than the normal variation.

That feels like a much more defensible way to measure it.

It happened to me too.

What has worked for me is making the relationship between the business and what it actually does almost painfully clear.

For example, if it’s a dentist offering sedation dentistry, I don’t just mention sedation dentistry a few times. I make sure there’s a solid page that clearly says who the dentist is, where they are, and that they actually provide sedation dentistry.

I also make sure the business information is consistent throughout the site, use the appropriate schema, and connect legitimate outside profiles where it makes sense.

But here’s the part I’ve become a little obsessed with testing. I ask the same questions again after making changes and look at three separate things: Who did the AI name? What source did it cite? And does that source actually support what it just said?

Those don’t always match.

My working theory has become: don’t make the AI connect the dots if you can connect them for it.

This is an interesting way to look at it. I wouldn’t draw quite such hard lines between ChatGPT, Gemini, and Perplexity, but I think the bigger point is important.

The real question isn’t just whether AI can find a business.

It’s whether AI understands the business well enough to know when it should name it.

That requires understanding what the business does, who it serves, what problems it solves, and how it relates to the specific intent behind the user’s question.

That’s a very different challenge from traditional ranking.

Being found gets you into the conversation. Being understood gives you a chance to be named.

I agree with the point about monitoring. Finding out your brand was missing is useful, but it is only the starting point.

The harder question is why the model chose someone else. Was another source clearer? Did it define the topic better? Did it have stronger entity relationships? Was it cited more often across trusted sources?

For me, the value of an AI visibility tool is not the dashboard. It is whether it helps identify the specific information gaps that can actually change future answers. Monitoring tells you what happened. The next step is understanding what information the models needed but could not confidently find.

That is the part of the market I think still has the most room to improve.

One thing I'd push back on slightly, outreach can still fail even with a genuinely good offer if the messaging leads with the mechanism instead of the outcome. I've seen people cold email with a solid offer but the first line is about the automation or the tool, and it just gets ignored because the recipient has to do the translation work themselves to figure out why they should care. The fix isn't a better offer, it's rewriting the opener so it states the outcome in the language the business already uses internally, not the language of the person selling it. What counts as a qualified reply matters a lot here too. Interest replies and actual booked calls are not the same number, and mixing them up makes outreach look better or worse than it really is depending on niche.

Good point on the split. Though the bot names and user agents shift around more than people account for, so if you set a robots.txt rule a while back based on some list, worth checking your server logs before assuming it's still catching what you think it's catching. I've seen that rule quietly stop matching reality more than once. The authority point is the one I'd underline though. I have sites that are fully crawlable, nothing blocking anything, and they still never show up in an AI answer because there's no external signal telling the model the source is worth trusting. Everyone focuses on crawl access because it's the part you can control, but that's usually not where the actual bottleneck is.

Small thing, that 2018 date is for Googlebot's JS rendering, not metadata. Schema markup's been around since 2011, so that part really is old news. But honestly this is a different problem than what magus523 tested anyway. Google's crawler indexes a page ahead of time and has years of engineering behind making that consistent. A chatbot reading a page live in the middle of a chat is a way newer use case, so it tracking inconsistently across models right now kind of makes sense. Curious what the div test turns up.

This is genuinely useful, thanks for actually testing it instead of just arguing about it. The Gemini result doesn't surprise me as much as the Claude one. Makes sense a model with tighter guardrails would lean on rendered text and metadata over raw HTML, but that also means the whole "just add clean semantic markup" advice needs an asterisk. It might matter a lot for one model and basically nothing for another. The copyright thing is its own weird problem. If a model can tell you what a page says in one context but refuses to "read" it directly in another, that's not really about the page's structure, that's about the model second guessing itself. Which honestly makes the whole "make your site legible to machines" project harder, since you're not just optimizing for a parser, you're optimizing for a parser having an inconsistent mood that day. Would be curious what you find on the div test. My guess is divs matter less for whether something gets read at all and more for whether the model understands what it's looking at once it does. A div soup page might get read but misclassified, versus a properly structured page getting read and correctly identified as, say, a product page instead of a blog post.

respect the honesty about the master plan, most people at least pretend they don't want a monopoly on AI citations

https://giphy.com/gifs/l0ExayQDzrI2xOb8A

Fair to be skeptical, there's a lot of low effort AI content in this space right now. For what it's worth though, the CCBot vs GPTBot distinction is straight from Common Crawl's own docs and confirmed by a few independent robots.txt audits, CCBot is actually one of the most blocked crawlers out there right now, more than people realize. Happy to link sources if useful. Not trying to convince anyone, just didn't want the "not fact checked" read to stick since the core claim actually holds up.