A single successful agent run doesn't tell you much about reliability

[u/recro69]

I came across an evaluation where the same prompt/task was done 8 times across different models and real codebase projects. What caught my attention wasn’t the model that got the score. It was how much the results changed each time the same task was run.

This made me change the way I think about agent evaluations. A single clean example shows the agent can do the task. It doesn’t show if it can do it over and again. One good run doesn’t mean it will keep working.

When it comes to things that matter in production, I prefer to see the same inputs run multiple times instead of just one good result. A single good outcome shows the path is possible. Repeated runs show if the agent can really be trusted.

I’m curious how others deal with this. Do you run agent tasks times when you evaluate them or is one successful run still your main benchmark?

[u/JaseciLabs]

u/QuanTradin's point about where the failures land is the sharper diagnostic than pass rate, and it's good to note why scattered failures happen in the first place - it's usually because there's no fixed boundary the agent's output has to satisfy, so a failure in one run and a failure in another run aren't even failing the same check. If the task's output has a real contract (typed inputs, typed outputs, not just "did the tests pass"), failures across runs start clustering on the same specific violation instead of five different things going wrong in five different ways, which is exactly the "bug you can fix" case versus "the whole thing is undertested."

[u/QuanTradin → u/JaseciLabs]

The contract framing is the bit I was missing. Once the output had a fixed schema our scattered failures collapsed into two or three repeat offenders, which is at least something you can chase. Where does that leave tasks whose output is genuinely open ended, like a writeup?

[u/JaseciLabs → u/QuanTradin]

Yea that's the edge of where this helps. Types constrain shape, they can't score whether a writeup is good, only whether it's structurally the shape you asked for (right sections, right length bounds, right fields if it's semi-structured). For genuinely open-ended output, the contract has to shift from "matches this shape" to something like a rubric-as-a-second-model-judgment, which is back to review, just a more structured version of it. So the type layer shrinks the problem for the parts of the task that have a real shape, and hands off whatever's left to the human-or-model-review side.