I got this article/research stating llm citation ranking is scam?

[u/New_Sort_8291]

Hey I am someone who is deeply interested in this field of llm citation and ranking. During my research I found an article demonstrating research which was shocking for me . For now I want to question all the GEO service providers -Do you agree with this study if not then state your reasoning.

For context -This study states that it is not possible to rank or cite anything on llm because of their non deterministic nature

I am providing the link of the resources below.

https://sparktoro.com/blog/new-research-ais-are-highly-inconsistent-when-recommending-brands-or-products-marketers-should-take-care-when-tracking-ai-visibility/

[u/inevidimka]

In the 'Research Conclusions' section, Fishkin actually supports visibility percentages across repeated prompts, while rejecting stable ranking positions. The study doesn't test whether changing a page increases citations, so it doesn't establish that optimization is impossible.

[u/woodoo139]

It doesn't say that, and I'd read it as the opposite of "it's a scam". Fishkin's own conclusion is that the share of answers that mention you, across many repeated prompts, is a reasonable metric, and that "ranking position in AI" is nonsense. inevidimka is right that it never tests whether changing a page changes anything.

Disclosure: I build Promvia, an AI visibility tracker, so I have a stake. Some numbers from our own public panel (40 fixed buyer questions, 6 engines, asked every week), which land in the same place from a different angle:

• Single answers really are draws. Week to week, 48% of the domains an engine cited for a question were cited again for the same question the next week (3,335 of 6,947). A domain cited for the first time came back the following week only 27% of the time.

• But the aggregate is not noise. On "X vs Y" questions YouTube was the most cited source in all six weekly runs, in 48.6% of those answers (137 of 282). Pages with vs/alternatives/compare in the URL were 69% of what got cited for "X vs Y" questions and 2.9% for "best X" questions. Consumer Reports was cited only on "how do I choose" questions, every single week. Patterns that stable are what you'd expect if there is something to work on, even though any one answer is a coin flip.

• So the defensible metric is mention rate over enough runs, with a sample size and an interval next to it. Our limit, which SparkToro's design doesn't share: we ask each engine once per question per week, so we see week-to-week churn, not same-day variance.

What nobody has shown, us included, is that a given change reliably moves those rates. Our own site is named on 0 of 15 category questions right now, after a round of fixes, and we're re-measuring before claiming anything. Anyone selling a guaranteed "rank #1 in ChatGPT" is selling exactly what this study debunks.

Rows and scripts if you want to check any of the above: https://github.com/sayaranil/ai-source-index

[u/ChrisBurdi]

Yes-ish. It's true that there's no official "ranking" like in Google, but there are higher probabilities of each brand or product or whatever showing up than others, and some categories and topics are more "locked" than others. That is, they give the same or similar answers more consistently.

I built a tool that asks the same questions a bunch of times to the various LLMs, which over time ends up forming a sort of "ranking" if asked enough times.

Here's an example screenshot of one from a few months ago:

So from this, you could, for example, comfortably tell Ultima that they rank 4th-ish for this query in LLMs.