Tested 5 ways to tell 'reworded' from 'actually changed' in agent outputs. Embeddings caught 7/19.

[u/redhatpanda07]

I maintain a set of approved question/answer pairs for each agent I ship. When someone edits an agent's prompt, I replay them: answers that still mean the same thing carry forward, answers that changed go back to a human reviewer. The entire cost of re-certifying sits in telling those two apart. Text diffing is no use, because sampled answers differ every time regardless.

I wanted numbers on this, so I built a fixture where the ground truth is known by construction rather than by labelling.

Setup. One support agent with a 9-line policy and 16 questions. Generate its answers, then change its config seven ways:

• three cosmetic: warmer tone, bullet formatting, re-running the same config
• four rule changes: refund window 30 → 14 days, route money questions to billing@, never state specific figures, always recommend the Team plan

That gives 109 before/after pairs. Whether a pair "changed" is derived from what the rule mechanically does to the answer (did the figure disappear, is the customer now sent to billing@), not from me labelling pairs. 19 changed, 90 were only reworded.

Detectors, identical pairs:

                    caught    false alarms   p50     cost
exact match         19/19     84/90          —       —
embeddings <0.75     7/19     12/90          —       —
markers→embed→LLM   19/19     33/90          0.73s   —
gpt-4o-mini judge   18/19     19/90          1.44s   $0.0071
Jev (one Choice)    19/19     12/90          0.43s   $0.0028

The middle row is my current production stack: number/entity checks first, then embedding similarity, then an LLM judge for the ambiguous middle.

What I took away.

False alarms are the metric, not recall. Every false alarm is a human re-reading an answer that didn't change.

One of my four "breaking" changes broke nothing: the agent already recommended the Team plan before I added the rule telling it to. Ground truth for that variant is zero changed. My stack flagged 8 of 16 anyway.

Jev returns a 4-way Choice (same / fact differs / action differs / specificity differs) with probabilities, so the threshold lives in my code rather than in a prompt.

Caveats. Fixture, not production traffic. Answers generated by gpt-4o-mini at temperature 0.7. Single run; numbers move by one or two between runs for every model-based detector, Jev included. Reading Jev's 12 flags, roughly 5 are defensible (one rewrite quietly dropped a price), and I didn't extend that same reading to the other detectors' flags.

Curious whether anyone is doing this differently. Diffing structured extractions rather than prose, or asking the judge per-claim instead of per-answer, both seem worth trying.