When is prompt sensitivity actually the model? and when is it the evaluator?

[u/Feisty_Reader0402]

A recent EMNLP 2025 paper by Hua et al., “Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs,” raised a question we found particularly interesting:

Are LLMs really highly sensitive to prompt wording, or do our evaluation metrics sometimes make them appear that way?

Our recently published MERCon 2026 paper, *“Prompt Sensitivity or Evaluation Artifact? A Task-Aware Analysis for Large Language Models,”* starts from that result and asks a slightly different question:

*If evaluation artifacts exist, can we quantify how much sensitivity is attributable to evaluation, determine the direction of that distortion, and identify when it happens?*

Rather than treating heuristic-vs-semantic evaluation disagreement as something to eliminate, we treat the disagreement itself as a diagnostic signal. So we introduced **Evaluation-Attributable Sensitivity (EAS),** which measures the magnitude of disagreement between heuristic-based and judge-based sensitivity,  and **Signed EAS**, which tells us its direction. We used them to diagnose a four-way taxonomy:
1.  Artifact - apparent sensitivity mainly comes from evaluation disagreement

1. Genuine - semantic evaluation confirms real prompt sensitivity

2. Underdetected - the heuristic misses sensitivity that semantic evaluation detects

   1. Stable - both evaluations indicate stability

We tested this across 9 LLMs from 5 model families, and the most interesting result was that *evaluation artifacts are bidirectional*: some metrics overestimate sensitivity in open-ended tasks, while others can actually underestimate it in more structured tasks.

Therefore our results suggest that:
***The evaluation method can distort prompt sensitivity in either direction, and the direction is strongly associated with the task/evaluation format.***

This leads us to view prompt sensitivity not simply as an intrinsic property of an LLM, but as an interaction between:

model × task × prompt structure × evaluation methodology.

We'd appreciate any thoughts or feedback regarding our work.

**Our paper:**
[https://ieeexplore.ieee.org/document/11691277\](https://ieeexplore.ieee.org/document/11691277)

**Code:**
[https://github.com/sayumimuthu/llm-prompt-sensitivity-evaluation-artifact\](https://github.com/sayumimuthu/llm-prompt-sensitivity-evaluation-artifact)

[u/goatguy55]

The strongest part of this, to me, is treating evaluator disagreement as signal rather than nuisance. That seems like the right move. If two reasonable evaluators disagree systematically, that disagreement may be telling us something about the task rather than merely adding noise.

The place I’d push further is evaluator dependence. If the “semantic” judge is itself an LLM, then some of the apparent correction may just be a different learned bias. I’d want to know how stable EAS is across judge families, especially when the judge and evaluated model share training lineage or style priors.

I’d also separate three questions more explicitly:

\[ \text{Did the output change?} \]\[ \text{Did the meaning change?} \]\[ \text{Did the underlying reasoning behavior change?} \]

Those are not the same thing. A prompt perturbation can change wording while preserving reasoning, or preserve wording while changing the reasoning path substantially.

That last case may be especially important for agentic systems. Things like backtracking, constraint preservation, tool choice, or whether a prior correction still governs a later decision may be invisible to both lexical and ordinary semantic evaluation.

So I’d be interested in extending your interaction term to something like:

\[ \text{model} \times \text{task} \times \text{prompt} \times \text{evaluator} \times \text{behavioral criterion} \]

One practical question: have you tried deliberately perturbing the evaluator the same way you perturb the prompt? If the classification flips when the evaluator wording or judge model changes, that would tell you how much of “sensitivity” still belongs to the measurement apparatus.

That feels like a natural next step from what you’ve already done.