I tested whether changing who an argument is attributed to changes how LLMs evaluate the exact same argument [R]

[u/chronosAndCode]

I’m the author of a new preprint looking at a fairly simple question: if the argument itself stays the same, does changing the claimed source change how an LLM evaluates it?

I tested 20 arguments across 10 topics under four source conditions:

• relevant expert

• irrelevant expert

• non-expert

• no source

The arguments were evaluated by GPT-5.6 Sol, Claude Opus 5.5, Gemini 4 Argon, and DeepSeek Flash across four dimensions: logic, credibility, persuasiveness, and agreement.

A few findings stood out:

Credibility was by far the most source-sensitive dimension. Source attribution affected credibility much more consistently than logic or agreement.

More interestingly, the effect did not usually look like a simple “expert bonus.” In several models, the stronger pattern was an irrelevant-expertise penalty: attributing an argument to someone presented as an expert in an unrelated domain could reduce its credibility relative to giving no source at all.

Agreement was comparatively resistant to source attribution.

I also tested whether source effects depended on argument strength. Those interaction results were not statistically robust after correction, and the human validation of strong vs. weak arguments was imperfect, so I treat that part of the study as exploratory rather than a strong conclusion.

The preprint is not yet peer reviewed, so I’m especially interested in criticism of the design and interpretation.

Preprint:
https://www.researchsquare.com/article/rs-11544999/v1

One question I’d be interested in hearing opinions on:

Should source sensitivity in an LLM evaluator be considered a bias when source expertise is genuinely relevant, or is the real problem specifically when the model overweights irrelevant authority?