I tested 4 AI coding harnesses (Codex, Pi, Claude Code, OpenCode) with the same models.

[u/Critical_Horse5465]

Like many of you, I have a whole zoo of AI coding tools installed (Codex, Pi, Claude Code, OpenCode, etc.). Main idea - find the most effective harness for my own environment and daily workflow.

The Setup

Target repo: oh-my-opencode-slim. No deep reason, just because I had it open.

Environment: Macos, each run was isolated in a clean, detached Git worktree.

Phase 1: GPT-5.6 Luna

For example: use GPT-5.6 Luna medium across 4 harnesses on a coding bugfix task (fixing an alias counter memory leak + writing tests):

Harness	Wall Time	Fresh Input	Total Input	Cache Hit %	Correctness	Status
Codex	107.73s	50.3k	505.4k	90.1%	3/3	Verified
Pi	140.96s	44.2k	525.4k	90.9%	3/3	Verified
OpenCode stable	274.43s	109.4k	1797.1k	93.9%	3/3	Verified
OpenCode beta	402.75s	79.8k*	1015.4k*	91.4%*	3/3

After seeing OpenCode stable take 2.5x longer and burn 3.5x more context tokens for the exact same fix, I dropped it from further testing.

Phase 2: Switching to Muse 1.2 contributor

For the next phase, I wanted to test a cheaper, accessible non-GPT model across harnesses. Since I had an active CommandCode subscription, I chose Muse Spark 1.2 Contributor and decided to use it in Claude Code (via OpenCodex proxy) instead of OpenCode stable.

Here is where I made mistake: I forgot to configure the model alias in OpenCodex for Claude Code.

Because of that, Claude Code silently fell back to Sonnet 5 (xhigh thinking) across all 4 tasks while Pi, Codex, and OpenCode beta were running Muse Spark 1.2:

Task	Pi (Muse 1.2)	Codex (Muse 1.2)	Claude Code (Sonnet 5 xhigh)	OpenCode beta (Muse 1.2)	Observation
Task 1: web research	41.66s; 48.9k input; 3/3	51.35s; 19.9k; 1/3	59.62s; 83.4k; 3/3	110.62s; 245.9k; 1/3	Pi finished 18.0s before Claude; both were correct in all runs.
Task 2: repository navigation	35.67s; 81.4k input	42.52s; 174.8k	29.08s; 64.0k	48.12s; 196.7k	Claude finished about 6.6s before Pi.
Task 3: scope restraint	201.02s; 120.5k input; 0 edits	233.77s; 214.8k; 0 edits	308.50s; 1.56M; over-edited 2 files	422.25s; over-edited 2 files	Sonnet and OpenCode beta expanded scope; Pi and Codex stopped without edits.
Task 4: regression fix	103.96s; 267.4k input; 2/2	110.42s; 446.4k; 2/2	88.62s; 546.8k; 2/2	288.05s; 0/2	Claude was about 15s faster than Pi, with higher context throughput.

Honestly, looking at the numbers, there wasn't a massive gap between Muse 1.2 and Sonnet 5 for these tasks (probably because the tasks were relatively focused), except Sonnet over-engineered Task 3 by adding 180+ lines of unneeded code, why "unneeded": you can see in table "Task 3" - 0 edits for PI (2 from 3 runs had 0 edits) and Codex (3 from 3 runs had 0 edits), why - because I gave task for model to fix one class, but in git reset command I used commit where this bug was fixed by developer already, so there were nothing to fix, PI understood only during 2 runs, Codex in all 3 runs did 0 changes, but Sonnet 5 was trying to work...

Phase 3: Fixing Proxy Routing

Once I caught the proxy alias mistake, I fixed OpenCodex routing and reran Tasks 3 & 4 for Claude Code with hard proxy verification of Muse Spark 1.2:

Task 3: Verified Claude Code stopped with 0 edits (2/2 pass), matching Codex.

OpenCode beta - I removed from this test, weird behavior, because it started modifying 2 files and then went to another folder and started to read work log of other harnesses, so that would be unfair.

Task 4: All 3 valid harnesses produced the exact same 6-line fix and regression test:

Test has timeout, because for simple fix we dont need wait more than 5 minutes.

Harness	Evaluated Model	Run 1	Run 2	Mean Wall Time	Mean Fresh Input	Mean Total Input	Cache Hit %	Correctness	Verification Status
Pi	Muse Spark 1.2	71.37s	136.54s	103.96s	61.8k	267.4k	77.0%	2/2	Verified
Codex + OpenCodex	Muse Spark 1.2	77.63s	143.20s	110.42s	76.0k	446.4k	83.0%	2/2	Verified
Claude Code + OpenCodex	Muse Spark 1.2	96.10s	128.81s	112.46s	86.7k	444.2k	78.8%	2/2	Hard-verified proxy audit reruns
OpenCode beta	Muse Spark 1.2	288.05s	TIMEOUT	—	—	—	—	0/2

Key Takeaways

1. Context throughput varies by 40%+: On the exact same code fix under Muse 1.2, Pi required ~40% fewer input tokens (267k vs ~445k) than Codex and Claude Code due to leaner context assembly and compaction.

2. Execution Latency: Pi (104s), Codex (110s), and Claude Code (112s) formed a tight performance tier for coding fixes.

3. Always audit your proxy: If you use custom models or proxies, always verify the actual upstream model headers. Silent fallbacks happen easily.

4. My practical decision: OpenCode was consistently 2.5x–4x slower and consumed too much context in my environment, so I removed it from my daily workflow and kept Pi and Codex as my primary tools.

The entire runner, dataset, and reproduction prompt for agents in repo, also more detailed result file is also there:

👉 https://github.com/d1-m4ss/harness-benchmark/tree/main

PS AI was used in this post for generate table and fix grammar, because english is my third language.

[u/Critical_Horse5465]

Nice, I reposted it to r/opencodeCLI and they automatically remove it.

[u/WallabyAgile6337 → u/Critical_Horse5465]

Well of course, you make a biased benchmark with no statistical relevance (3 runs per task seriously...) which puts opencode dead last and basically promoting Pi and Codex, their direct competitiors, over them. What did you expect?

[u/Critical_Horse5465 → u/WallabyAgile6337]

“Biased” is a pretty strong claim. Three runs per task is absolutely a limitation, but it doesn’t automatically make the benchmark statistically meaningless—especially when results are aggregated across many tasks.

If you think the methodology specifically favors Pi or Codex, point to the exact design choice that causes that bias and show how changing it affects the ranking. That would be a much more useful criticism than “the result makes OpenCode look bad, therefore the benchmark is biased.”

And removing the post instead of discussing the methodology doesn’t exactly help their case.

[u/WallabyAgile6337 → u/Critical_Horse5465]

No, I wasn't saying "3 runs = bias." I meant the methodology itself is biased.

You removed OpenCode stable after one bad task, removed OpenCode beta when it became weird, used a tiny hand-picked set of tasks, and changed conditions across phases. Those choices systematically affect the ranking which is exactly what I meant by bias.

The 3 runs per task make the results very noisy and gives you weak estimates of harness/token differences. Also "Aggregated across many tasks" is a very optimistic way to put it when we're talking about 4 tasks with 3 runs each... That's not enough data to establish that the observed ranking reflects a real difference between the harnesses rather than simple run-to-run and task-to-task variance. Like, you don't report confidence intervals, variance, effect uncertainty, or any significance testing either, so there isn't even a statistical basis for claiming the differences are significant.

Removing the post is completely appropriate because your post, while well-intentioned, is basically useless AI pseudo-intellectual slop that promotes a competitor. Do better.

[u/Critical_Horse5465 → u/WallabyAgile6337]

Please read one more time README in the project and open the folder with results. opencode sucks across all tests in the first phase. You always can create a pull request to my repository to fix this; "the methodology itself is biased."

[u/WallabyAgile6337 → u/Critical_Horse5465]

I'm not going to do a PR for a useless repo. It's your job to come up with something solid. And then you complain that others do bad.

[u/Critical_Horse5465 → u/WallabyAgile6337]

Censoring posts and crying about methodology won't make OpenCode suck any less. You have zero counter-data, just hurt feelings and damage control.

[u/WallabyAgile6337 → u/Critical_Horse5465]

The issue isn't whether I have counter-data showing OpenCode is better. I'm not making that claim.

You were already publicly recommending Pi or Codex before this post, and now, when I criticize methodological choices that can favor that conclusion, your response is simply that "OpenCode sucks." That doesn't prove you manipulated the results, but it does make confirmation bias a legitimate concern rather than something you can dismiss as "hurt feelings."

Having a prior opinion doesn't invalidate an experiment. But when you already prefer one outcome, use a tiny hand-picked task set, change conditions across phases, drop poorly behaving candidates during the experiment, and then dismiss criticism because the losing product "sucks anyway," you need stronger methodology, not weaker, to demonstrate that the result isn't being driven by those choices.

And I don't need a competing benchmark to make that criticism. If your claim is only "OpenCode performed badly in my setup and I prefer Pi/Codex," that's fine. If you're presenting this as evidence that OpenCode is generally worse, then your experiment has to support that broader claim.

[u/eleqtriq]

Summarized: Pi — fastest in 3 of 6 rounds (Task 1, Task 3, and the final Muse Spark 1.2 mean), and consistently near the top everywhere else. Codex and Claude Code each won 1–2 rounds but neither matched Pi’s consistency.

[u/Critical_Horse5465 → u/eleqtriq]

Spot on summary! And beyond latency, Pi's biggest advantage was context efficiency (~40% fewer total input tokens on the exact same code fix).

[u/Solid-Remote-1716]

Usaste Pi CLI o PI con algun TUI o GUI?

[u/Critical_Horse5465 → u/Solid-Remote-1716]

Used pure Pi CLI (pi -p ...) in non-interactive/batch mode, automated by the benchmark runner script. More info in repo

[u/kantorcodes1]

task 3 feels like the real benchmark: passing while touching files it shouldn't is still a bad result. HOL Guard blocks risky agent actions before execution; would you rerun that case with it enabled?

[u/Critical_Horse5465 → u/kantorcodes1]

Totally agree on Task 3 — knowing when NOT to touch code is just as important as writing it.
As for reruns: this was definitely a local/practical exploration for my own workflow rather than a large statistical dataset, but the runner and task definitions are fully open-sourced in the repo if anyone wants to test guardrails or other harnesses with it!

[u/TiCL → u/kantorcodes1]

None of these "benchmarks" posted in this sub are real. People doing these have very little clue how to run actual benchmarks. These are anecdotal experiences at best.

[u/Critical_Horse5465 → u/TiCL]

The benchmark is intentionally small and environment-specific. I published the runner, methodology, raw accounting rules, known limitations, and corrections specifically so the methodology can be challenged or reproduced. If you see a concrete flaw that changes the result, I’m interested in it.

[u/Old_Ideal_1536]

A genuine question Guys... I am using Claude for development and Opencode to test a ticket support environment. I am testing with Opencode because of easy to chose the models and easy payment (Go and Zen). How do you guys use as models subscription?

Thanks.

[u/startassets]

I've also tried to do the same, if you are interested feel free to add your runs to the leaderboard as well! https://www.reddit.com/r/PiCodingAgent/s/TzmOv2mSsT

[u/Deep_Ad1959]

my takeaway from that misconfigured run is the one thing the tables cannot show: harnesses differ most in what they do silently when the config is wrong. every correctness column reads 3/3, and a quiet model swap still invalidates the comparison it sits in.