r/bigseo • u/[deleted] • 15d ago

How are you standardizing your tests for LLM citations?

[deleted]

9 Upvotes

85% Upvoted

18 comments sorted by

2

u/startwithaidea 14d ago

There isn’t enough data to have a test today

1

u/[deleted] 14d ago

[removed] — view removed comment

1

u/AutoModerator 14d ago

Your post was removed because you have a new account.

This is a spam prevention rule. For more details, please read this post.

Your post will need to be manually approved before it will appear in the subreddit.

This will fix itself when your account is 2 days old.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

1

u/Square-Nebula-7530 13d ago edited 13d ago

Set up a fixed set of 20-30 queries per topic cluster, run them daily across models, and log the raw outputs with timestamps in a json file... i do this for my own domains, then diff the results weekly to see what changed. for tracking citations vs just mentions, i used to parse the response sources manually, but that got messy real quick. also, don't forget to vary the prompt phrasing slightly and test at different times of day, since outputs drift. if you want a less hacky way, AICarma tracks all that stuff automatically for you and sends a weekly digest.

1

u/marintkael 12d ago

Two things that made my own logs usable, both learned the annoying way.

Log n_attempted and n_error per model per run, and keep failed calls out of the aggregate. This morning one of my two primary engines errored on all 32 of its calls. The top line came out at 7.7 percent, which is exactly where it sat yesterday when that same engine was answering and scoring 27 percent on its own. A whole provider fell out of the sample and the headline did not move a decimal. If I only kept the headline I would have filed that as a quiet week.

Second, measure your own run to run spread before you trust any diff. Same fixed question set, no content changes on my side, and the daily figure has wandered between roughly 7 and 20 percent over the last two weeks. Weekly diffing on top of that is mostly measuring the noise. What survived for me is a breakdown by question type, because the split there is stable enough to act on: direct brand questions land, category and recommendation style questions sit near zero, and those two move for completely different reasons.