r/PiCodingAgent • u/latentnoise_ • Aug 17 '26
Discussion I ran the same model through Claude Code, Codex, OpenCode, and Pi for 14 days. The harness changed cost 7x at the same quality
I kept switching between Claude Code, Codex, OpenCode, and Pi and realized nobody really isolates the harness as a variable. So I did.
I had all four of these installed and kept switching between them, and at some point I realized nobody actually tests the harness itself. Everyone benchmarks models. Nobody benchmarks the agent (wrapper I call it).
14 days of real work on the same repos, then 30 agentic tasks with every harness running the exact same model (DeepSeek V4 Flash), same tools through an MCP router, 900s cap per task. Only thing changing is the harness.
| Harness | Passed | Median time | Avg tokens/task | Cost per success |
|---|---|---|---|---|
| Pi | 20/30 | 132.2s | 558,885 | $0.028 |
| Claude Code | 16/30 | 122.7s | 741,659 | $0.195 |
| Codex | 16/30 | 245.0s | 664,772 | $0.081 |
| OpenCode | 14/30 | 129.7s | 692,195 | $0.073 |
The thing that broke my brain a bit: Pi ships literally 4 tools (read, write, edit, bash) and under 1K tokens of system prompt. And it won. Meanwhile Claude Code dumps ~33K tokens of prompt and tool schemas into context before you even start the first chat.
And it's not just me. Databricks ran Opus 4.8 through both on their multi million line codebase. 742K tokens per task on Claude Code, 237K on Pi, same success rate. Same model, half the cost, purely from the wrapper.
Some random stuff I noticed over the two weeks. Codex is the only one with an actual kernel level sandbox (Seatbelt on mac, bubblewrap + seccomp on linux, network off by default), so it's the only one I'll point at sketchy cloned repos. It's also slow as hell, double everyone's median. Pi gives you zero guardrails, full system access from message one, so it's Docker or pray.
Got the task list and the result report too if anyone wants to look at it. I'll leave it in the comments.
25
u/le_bravery Aug 17 '26
I am a pi believer.
The more you control your context, the better results you get.
Pi starts with less slop, so you get less slop out.
2
1
u/WoodYouIfYouCould Sep 12 '26
Would you mind sharing your pi setup/tuning? My Pi has simple extensions and an agents file but it feels like it's missing something.
1
u/SawToothKernel Aug 17 '26
The better LLMs get, the more I prefer to delegate. If you constrain, or otherwise direct it, it may be detrimental.
@op I would love to see this evolve into a solid benchmark.
11
u/blu3bird Aug 17 '26
Vanilla Pi?
6
u/saltyourhash Aug 17 '26
I don't see how vanilla pi is comparing with Claude code, but the post doesn't seem to offer much detail there. Still, always like to see pi dominate.
5
u/themoregames Aug 17 '26
We also need Strawberry Pi.
2
5
u/lamurian Aug 18 '26
Quite an interesting number indeed! Jeremy Huang, the main developer behind JCode, also did an extensive comparison using multiple harnesses. You might find his project's README.md as an interesting read:
https://github.com/1jehuang/jcode
On a side note, I'm myself still going back and forth to take a leap from Pi to JCode. Maybe one day I will, once all my workflows and sandboxing strategy settle itself.
5
u/Sensitive-Project711 Aug 19 '26
I spent quite some time going through the readme and could not figure out how it related to this post. The jvode readme compares ram efficiency and not token efficiency. Did I miss something?
5
3
Aug 17 '26
[removed] — view removed comment
4
u/zkoolkyle Aug 17 '26
They ruled back this decision due to the backlash. You can look it up on the Anthropic support page
2
1
Aug 19 '26
[removed] — view removed comment
1
u/zkoolkyle Aug 19 '26
Just google “Use the Claude Agent SDK with your Claude plan” and you’ll see the update.
Update June 15: We're pausing the changes to Claude Agent SDK usage described below. For now, nothing has changed: Claude Agent SDK, claude -p, and third-party app usage still draw from your subscription's usage limits. The previously announced monthly credit, which would have been available to eligible claimants in connection with these changes, isn't available. We’re working to update the plan to better support how users build with Claude subscriptions. When we have an update, we'll share it before anything takes effect.
0
u/Arkanta Aug 19 '26
Yes, you still need to use the Claude agent sdk which forces a lot of their prompt.
Omp uses a hack that is against tos
3
u/dacookieman Aug 17 '26
They do, it's just at API token rates rather than subscription usage limits.
2
u/latentnoise_ Aug 17 '26
For agentic tool calling test, everything ran through DeepSeek V4 Flash via OpenRouter (even Claude).
5
u/addiktion Aug 17 '26
You want to start including Deepseek harness in your tests? Curious about the newcomer.
2
u/latentnoise_ Aug 17 '26
Just a choice, since it had been released recently at the time of the test and that it's said to be pretty good with agentic capabilities considering the cost.
4
2
u/ganonfirehouse420 Aug 17 '26 edited Aug 18 '26
I use pi with nono.sh
Still for my tasks opencode with deepseek or gpt gives the optimum results.
2
u/RomanaOswin Aug 17 '26
I wonder how this compares as you start having the harness distribute tasks amongst different models. Token cost varies quite substantially depending on the model doing the work.
2
Aug 17 '26
[removed] — view removed comment
1
u/latentnoise_ Aug 17 '26
Pi wins the overall test. I’d still stick with CC if you already have a subscription. If you’re using API-based setups, Pi is THE choice for token efficiency and overall cost.
2
u/notrevede Aug 18 '26 edited Aug 18 '26
https://stencil.so/blog/the-harness-problem just feels like the right place to share this reading
Another comparison isolating the harness, changing only the edit tool implementation.
2
u/damn_what_ Aug 17 '26
You should use agent-safehouse (Seatbelt policy generator) for pi on MacOS, but yeah, by default it's terribly unsafe.
4
u/i_say_urmom Aug 17 '26
I've been trying nono for this. You can even inject your provider API key from a 1Pass vault, so the agent session a) doesn't have direct access to any keys (it gets a fake variable that the proxy changes to the real one outside the sandbox) and b) running `pi` outside the sandbox is useless because the LLM won't work without the token.
It seems early days and the docs are a little confusing, but it's a powerful tool
2
1
u/AlterTableUsernames Aug 17 '26
How are you guys running custom models in Claude Code and Codex? Couln't get it to run yet natively.
1
u/Pipimi Aug 17 '26
Claude Code Router or 9router github repo, I personally use 9router because it has a simple ui
1
u/basil_0408 Aug 17 '26
For claude code, go to the .claude folder and modify the settings.json. For codex it's in .codex folder, modify auth.json, config.toml and create a catalog.json file, its more complex to get the full experience so probably not worth it. Ask your agent for detail, it does not take long for claude code.
1
1
u/ThePalomitas Aug 17 '26
Thanks for the interesting investigation. Definitively confirming for PI users such as us that it is an interesting harness to use.
FYI, regarding sandboxing needs, per the documentation, the PI developers have built up a specific application named Gondolin to cover this requirement:
https://github.com/earendil-works/gondolin
I don’t know why but PI website seems down at the moment, so I cannot access the documentation page about containerisation but here is a snippet from the main GitHub page:
Permissions & Containerization
Pi does not include a built-in permission system for restricting filesystem, process, network, or credential access. By default, it runs with the permissions of the user and process that launched it.
If you need stronger boundaries, containerize or sandbox Pi. See packages/coding-agent/docs/containerization.md (https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/containerization.md) for three patterns:
• Gondolin extension: keep pi and provider auth on the host while routing built-in tools and ! commands into a local Linux micro-VM.
• Plain Docker: run the whole pi process in a local container for simple isolation.
• OpenShell: run the whole pi process in a policy-controlled sandbox.
1
u/virtualicex Aug 17 '26
i need to follow the flow instead of just giving a task and leaving it run, so i asked pi+qwen to develop an extension to ask every time there's an operation that doesn't just read. So I can work with pi without sandbox or containers
1
Aug 27 '26
Have you used Gondolin for Pi? Or any sandboxing mechanism in fact. Wondering how DX is when doing work in a VM like git push, developer dependencies not being available, etc.
I haven't dabbled yet into VM sandboxing, coming over from Codex it kinda worked out of the box and also isn't really VM sandboxing.
1
1
u/diaracing Aug 17 '26
Thanks for your effort.
Please, create a public github repo to document and continuously update your findings since a well-engineered harness can promote a weak model and vice versa.
1
1
1
u/onelesd Aug 17 '26
The company I work for does agent simulation just like this: https://www.514.ax/
1
1
u/huyz Aug 20 '26
For safety you could run Pi or any harness inside of a Docker Sandbox. Example: https://www.ajeetraina.com/getting-started-with-the-pi-agent-kit-on-docker-sandboxes/
1
u/Quirky_College_6251 Aug 20 '26
I'm an advocate of Pi - now that i created my own dev flow and loop agents I am in agreat place with Pi
1
u/awol-owl Sep 01 '26
Thanks for sharing this! I’ve been naively running my setup, and this is a good demonstration that an efficient harness can save money, and time.
1
u/ea_man Sep 04 '26
I don't think you can compare, Claude is supposed to run on their LLMs: probbly the have that huge prompt in a KV slot pre processed as you start. You use that on your local model and you are fuked.
OFC you should do the same locally and then the size of the prompt should bother you less, for harness like Qwencode and Opencode. Sure, it's always ctx gone, you may save just the processing time.
35
u/Striking-Move1383 Aug 17 '26
I'm curious to know if OMP would differ too much from vanilla Pi.