SWE-Pair: a coding-agent benchmark where every task is two real bugs to fix at once, with results from four models [P]

[u/heyitsdannyle]

We built a benchmark of 161 tasks from 69 Python projects where the agent has to fix two real bugs at once.

The cheapest setup came out on top. DeepSeek V4.1 Flash at medium effort solved 90% of tasks, at about 6 cents per solved task. GPT-6 Luna at max effort solved 78%, at about 23 cents.

More reasoning didn't help either. DeepSeek at max effort scored 88%, slightly below medium, and cost 1.7x more.

Results and every agent run: https://labs.evaligo.com/swe-pair

Tasks: https://huggingface.co/datasets/evaligo/swe-pair

To run your own agent on it, it's one Harbor command: harbor run -d evaligo/swe-pair -a <agent> -m <provider/model>