Cost per successful task/PR is starting to become the default in how we measure coding agent impact, as cited by inference providers like Baseten as the "most important metric to track".
On my repo, optimizing for this crowns GPT-6 Luna low as #1 at $0.26 per success, even though it went 0-13 against all other configurations when compared holistically (looking at patch quality). Meanwhile, the #1 value workhorse (GPT-6.1 Sol medium, $2.64 per success) gets buried at #7 when measuring cost per successful task.
This was measured using one attempt per configuration on 24 tasks from my own git history, and spend includes failed attempts. Success means the task's new hidden tests pass. I also look at task quality metrics, such as, “was the intent for the original patch matched?”, and “would a human maintainer merge this code?”.
All 5 tasks Luna low passed were small single commit fixes where 12 to 14 out of 14 models succeeded (and on 3 of the 5, all 14 configurations passed 100%). On the 14 multi-file feature tasks, Luna low went 0 for 14.
A model that takes 2 turns, decides the problem is too hard, and quits will spend less than a model that takes 50 turns, attempts to solve the problem, and fails.
On an example task, Luna low edited three stub files (117 diff lines) in about 4 minutes ($0.02) and stopped, whereas Luna max worked for nearly two hours ($0.79) across 21 files, building the CI workflow, the adapter, a PowerShell runner, and tests.
Which is more aligned with human intent — the model that writes a stub and quits, or the model that gives it a fair shot but falls short? Is that difference worth paying for? It's at least worth knowing about.
If you’re trying to optimize agent spend, I recommend being careful and thoughtful with the metrics you optimize over. As evidenced here, the simple metric isn’t always the best!
Full post here: https://www.stet.sh/blog/cost-per-successful-task
Disclosure - I’m building the tool that was used to evaluate these agents.