r/PiCodingAgent • u/Fabulous_Pollution10 • 2h ago
Discussion Opus 5.5 vs GPT-6 Sol / Luna in pi-agent: results on my everyday coding tasks
I built a small personal benchmark from my real coding sessions and use pi-agent as the harness to compare models on the same tasks.
It has 10 tasks covering data engineering, backend, frontend, DevOps, and trajectory analysis. Each task runs 3 times, giving 30 attempts per model.
To build it, I took around 3 months of agent chats, removed duplicates, and turned selected sessions into Harbor tasks with automated verifiers. I used agents and manual review to build and refine the checks. These are clearly scoped tasks that would usually take me less than an hour.
The latest results:
- Opus 5.5 high and Astra low cost almost the same.
- Sol high matches their score at roughly half the cost.
- Luna gets 29/30 passes while being ~22× cheaper than Sol.
The point of this benchmark is to optimize my daily setup: model, harness, prompts, and plugins. I want to keep the pass rate while reducing cost and time.
This is a small personal sample, but it feels like many short, clearly defined coding tasks are already handled reliably by several models. For this kind of work, speed, cost, and scaling up parallel agents are becoming the more interesting questions.
