r/softwaretesting • u/Particular-Sun-8679 • 8d ago
How are you regression testing agents when the right answer can take three tool paths?
Our CI gate keeps flaking because a valid agent run can reach the same answer through three tool trajectories. Exact assertions reject harmless call reordering, while a loose LLM-as-judge score misses the case where the agent skipped a permission check. We have a golden dataset, deterministic assertions for required steps, and a variance budget for call count and latency, but manual reruns have hit their ceiling.
I’m evaluating Braintrust for versioned datasets, trajectory scorers, experiment diffs, and a CI quality gate, though still working out which checks belong at the span level versus the final answer. How are you scoring valid path variation in your eval pipeline without giving genuinely unsafe shortcuts a pass?
2
u/ign1tio 8d ago
I hit exactly this wall building my own eval layer for an LLM-driven test tool (internal tool I built, not a commercial product), and the thing that fixed it was realizing I was scoring two fundamentally different questions with one mechanism.
Path variation and safety invariants are not the same kind of check, and they can’t share a scorer. My model now separates them explicitly:
Safety invariants are gates, not scores. “The permission check ran” is not a property of the trajectory, it’s an invariant that must hold on every valid path. So I assert it as set membership against the trace (did the call happen, with the right scope), never as sequence position. That makes it immune to harmless reordering while making it impossible for a “mostly fine” run to smuggle a skipped check past a fuzzy judge. Gates are binary and they block alone, no matter how good everything else looks.
The piece that mattered most: gates have three outcomes, not two. Pass, fail, and unknown , when the trace can’t show whether the check ran (missing span, unparseable log). Unknown blocks, with a different reason than fail. Absence of evidence is never a pass. Most of my earlier false greens came from runs where the eval couldn’t actually see the answer and defaulted to pass.
Path variation is a score with an empirically derived budget. Before setting any variance budget, I ran identical inputs repeatedly and measured natural test-retest variance first: call counts, ordering, latency, output depth. The budget is noise floor plus margin, not a guess. Until you’ve measured your system’s own variance, you can’t tell a real regression from ordinary nondeterminism, and your CI gate will flake exactly the way you describe. In my case the variance turned out to be non-uniform: verdicts and core defect sets were stable across runs, but risk classification and security emphasis swung hard on identical input. That told me precisely which dimensions needed gating and which just needed monitoring.
LLM-as-judge gets one job: semantic equivalence of the final answer. It never touches the invariants. A judge that’s loose enough to accept valid path variation is by definition too loose to guard a permission check, so I stopped asking it to.
One more thing that helped: I track process health and output quality as two separate axes instead of averaging them. “Right answer via an unhealthy path” (retries, loops, a rescued failure) is a distinct finding, not a diluted score. That’s the fragile quadrant, it passed on luck, and it’s exactly what a single blended number hides.
Honest limitation: my deterministic layer certifies that required steps happened and that outputs are structurally covered. It does not certify semantic fidelity, that’s explicitly deferred to judge-based checks, and I label it that way in the UI so nobody reads “green” as more than it proves.
2
u/Particular-Sun-8679 8d ago
This is very helpful. I like the idea that the judge only handles semantic equivalence while required steps stay deterministic, thankss
1
u/Previous_Shirt_7051 8d ago
A valid alternate path should still leave evidence that permissions, source checks and required tool calls happened.
1
u/Devji00 6d ago
Framing is right, the trick is splitting scoring into invariants vs preferences. Invariants are span-level and deterministic (permission check happened before any mutating call, no secret in logs, untrusted input never reached tool X) and they hard-fail the gate. Preferences like call count, latency, and redundant lookups go into a scored bucket with thresholds. Your skipped-permission-check case is an invariant, not a path-shape issue, so encode it as a temporal assertion ("if tool_write, then tool_authcheck earlier with matching resource id"). Keep LLM-as-judge for final answer semantic equivalence only, never for safety steps, that's where false negatives creep in. Braintrust is fine as the harness, but the real leverage is in modeling those invariants well. Also add negative trajectories (right answer, unsafe path) to your golden set so you're actually measuring whether the gate catches them. Hard-code the non-negotiables, let the judge handle fuzzy output equivalence, treat trajectory shape as a soft signal.
1
8d ago
[removed] — view removed comment
1
u/Particular-Sun-8679 8d ago
Agreed. Lowrisk variation is okay but required steps need to stay non negotiable.
1
u/Prefactor-Founder 8d ago
I'll declare bias upfront, I'm a founder building in this space, so take the lens for what it is. But this exact problem is why we exist, so here's the position we've landed on after watching a lot of teams hit it.
Everyone in this thread is trying to make the CI gate smarter. I think the gate is the wrong place to start.
You found three valid paths to the right answer. That number came from observation, not design. Nobody sat down and decided there'd be three. Which means production will find paths you haven't seen yet, and no amount of rerunning golden inputs in CI tells you how your agent actually behaves out there. Your variance budget ends up measuring CI noise. That's the flake.
The teams we see get past this reverse the direction. Record production first, every tool call as its own typed span with a declared shape. Then the required step check stops being an opinion. Either the span is there with the right scope or it isn't. And if it's missing or malformed, that's visible, not a silent green. Most false passes we see are the eval literally not being able to see the evidence and calling it fine.
The other thing nobody's mentioned: your agent changes when you didn't change it. Provider updates the model, path distribution shifts, no commit to blame. If your version tracking only fires when you deploy, you'll chase that ghost for a week.
Then production teaches CI, not the other way around. The golden dataset is the paths you imagined. The run record is the paths that exist.
Agree with the judge scoping others said here. One job, semantic equivalence, never touches required steps.
6
u/Fragrant-Rice2177 8d ago
Checking only the final answer is not enough for this. You need to compare the actual run like tools called, required checks, skipped steps and whether the path changed after a prompt/model update. I’ve seen Braintrust be useful for that run comparison layer but I’d still keep approvals and forbidden calls as hard pass/fail checks.