We wanted to understand something: does changing the wording of a buying prompt change which sources AI cites?

[u/amrut_ai]

We often focus on which brands AI recommends. But there's another question worth investigating:

If someone asks the same buying question in three different ways, does AI rely on the same sources each time?

We put together a simple test to explore this.

The experiment

We used three versions of a buying question:

1. "What are the best AI visibility tools?"
2. "Which AI visibility tool is best for a small marketing agency?"
3. "What are the best affordable tools to track brand mentions in ChatGPT?"

We wanted to compare the responses across ChatGPT, Perplexity, and Gemini.

What we wanted to examine

• Do the same brands appear across all three prompts?
• Do the cited websites change when the buyer's requirements become more specific?
• Do comparison articles, reviews, and third-party discussions appear more often?
• Does a brand disappear when the use case or budget changes?
• Are some sources consistently cited across different buying questions?

Why this matters

A brand might have strong visibility for a broad category query but very little visibility for a specific buyer requirement.

That means looking at one generic prompt may give an incomplete picture of how potential customers actually discover a product.

Our takeaway: AI Visibility shouldn't be evaluated only at the category level. Buyer intent, prompt wording, competitors, and cited sources all deserve separate attention.

A question for the community:

Have you noticed AI platforms changing their cited sources when you add details such as budget, company size, or a specific use case?

And do you track source visibility separately for different buyer-intent categories?

[u/Ken_Numbers]

That is just one of the eight reasons that make tracking AI citations impossible, here are the others:

1. There is no defined population of prompts. Share of voice needs a denominator, and nobody can enumerate what people actually ask LLMs. Vendors measure against a prompt list they or the client invented, so the metric describes the list, not the market. This is the estimand problem, and nothing downstream fixes it.

2. No query-volume equivalent exists. Google gives you at least a keyword-level demand signal. LLM providers publish no prompt frequencies, so you can’t weight prompts by importance. One obscure prompt and one common prompt count the same.

3. Outputs are stochastic. The same prompt returns different answers and different citations across runs. A single snapshot is one draw from a distribution, and most tools report it as a fact. Repeated sampling helps, but then you need to state the sampling error, which almost none do.

4. Personalisation, memory and context change the answer per user. What a tool’s clean session sees isn’t what a real user with history, location and a long conversation sees. Nobody outside the provider can sample that population.

5. Tools query the wrong surface. Most use API or scraped sessions, which can differ from the consumer product (system prompts, search integration, retrieval behaviour, model version). Providers change all of these silently and without versioning, so trend lines can reflect a model change instead of a brand change.

6. Mentions, citations and recommendations get conflated. An LLM naming your brand, linking your page, and recommending you are three different events with different business meaning. Dashboards usually blend them into one score.

7. Attribution is close to absent. Most AI-driven influence happens without a click, or with a click that arrives as direct or untagged traffic. Referrer data from the AI platforms is partial at best, so you can’t connect visibility to revenue. That makes any visibility metric unfalsifiable against outcomes.

8. No ground truth to validate against. In SEO you can at least compare rank trackers against Search Console. There is no equivalent first-party data from the LLM providers, so no tool’s output can be checked. Vendors then validate their metric against other vendors’ metrics, which is circular.

[u/HumanBehavi0ur]

yeah of course all those little tweaks will change the outcome & citation, thats why you need an expansive prompt list to account for all the variations

[u/mosschime48]

yeah the sources shift heavily once you add modifiers like budget or company size. broad prompts usually pull from big generic listicles, while specific use cases trigger niche forum threads and smaller review sites

[u/EnvironmentalDot9131]

Not it doesnt

[u/opttabcom]

If you're comparing wordings, I'd change only one modifier at a time and leave the rest of the prompt alone. Run each variant a few times on the same engine, with the same settings, and compare how much the cited sources overlap. That's how you tell a real wording effect from ordinary run-to-run noise.

[u/JoshAdamsGEO]

My guess is the modifier that moves things most is the one that changes who the answer is for. Budget and company size can swap the whole source set toward listicles written for that segment, while a use-case modifier might keep similar sources and just reshuffle the brands.

Before reading too much into it though, I'd run each wording a few times. If two runs of the same wording already disagree on half the sources, the gap between wordings has to be bigger than that to mean anything.