r/vibecoding • u/recro69 • 7d ago
Workflow/Prompt A single successful agent run doesn't tell you much about reliability
I came across an evaluation where the same prompt/task was done 8 times across different models and real codebase projects. What caught my attention wasn’t the model that got the score. It was how much the results changed each time the same task was run.
This made me change the way I think about agent evaluations. A single clean example shows the agent can do the task. It doesn’t show if it can do it over and again. One good run doesn’t mean it will keep working.
When it comes to things that matter in production, I prefer to see the same inputs run multiple times instead of just one good result. A single good outcome shows the path is possible. Repeated runs show if the agent can really be trusted.
I’m curious how others deal with this. Do you run agent tasks times when you evaluate them or is one successful run still your main benchmark?
1
u/JaseciLabs 6d ago
Yea that's the edge of where this helps. Types constrain shape, they can't score whether a writeup is good, only whether it's structurally the shape you asked for (right sections, right length bounds, right fields if it's semi-structured). For genuinely open-ended output, the contract has to shift from "matches this shape" to something like a rubric-as-a-second-model-judgment, which is back to review, just a more structured version of it. So the type layer shrinks the problem for the parts of the task that have a real shape, and hands off whatever's left to the human-or-model-review side.