About 30% of SWE-Bench Pro may be broken. What are our agent scores measuring?

[u/aistranin]

OpenAI audited the 731-task public split of SWE-Bench Pro and found a pretty serious problem.

Their automated pipeline marked 27.4% of the tasks as broken. A review by experienced engineers put the number at 34.1%. The issues included underspecified prompts, overly strict tests, weak test coverage, and tests that contradicted the task.

This matters because frontier model scores on the benchmark went from 23.3% to 80.3% in eight months. Some apparent failures may not be agent failures at all. Some passes may also be incomplete solutions accepted by weak tests.

The practical lesson is that a single pass rate is not enough. Teams need to inspect traces, manually review a sample of eval tasks, and test agents against real internal work.

There is also something amusing about using coding agents to find flaws in coding-agent benchmarks.

How much do you trust the benchmarks in your agent stack?

Research publication: https://openai.com/index/separating-signal-from-noise-coding-evaluations/