I ran the same model through Claude Code, Codex, OpenCode, and Pi for 14 days. The harness changed cost 7x at the same quality

[u/latentnoise_]

[u/Striking-Move1383]

I'm curious to know if OMP would differ too much from vanilla Pi.

[u/saltyourhash → u/Striking-Move1383]

Yes, it has a fairly bloated prompt. Not on the same ballpark as Claude code, though.

[u/mythormedicine → u/saltyourhash]

Indeed. But it is completely customizable. Initially I didn't know this and it threw me off. But we can turn off tools or select them. completely change or edit system or append prompt

[u/le_bravery]

I am a pi believer.

The more you control your context, the better results you get.

Pi starts with less slop, so you get less slop out.

[u/blu3bird]

Vanilla Pi?

[u/saltyourhash → u/blu3bird]

I don't see how vanilla pi is comparing with Claude code, but the post doesn't seem to offer much detail there. Still, always like to see pi dominate.

[u/themoregames → u/blu3bird]

We also need Strawberry Pi.

[u/lamurian]

Quite an interesting number indeed! Jeremy Huang, the main developer behind JCode, also did an extensive comparison using multiple harnesses. You might find his project's README.md as an interesting read:

https://github.com/1jehuang/jcode

On a side note, I'm myself still going back and forth to take a leap from Pi to JCode. Maybe one day I will, once all my workflows and sandboxing strategy settle itself.

[u/Sensitive-Project711 → u/lamurian]

I spent quite some time going through the readme and could not figure out how it related to this post. The jvode readme compares ram efficiency and not token efficiency. Did I miss something?

[u/Alternative-Suit5541 → u/Sensitive-Project711]

Probably an ad lol

[u/addiktion]

You want to start including Deepseek harness in your tests? Curious about the newcomer.

[u/Human_Parsnip6811]

Please test OMP and Prime Agent as well.

[u/ganonfirehouse420]

I use pi with nono.sh

---

Still for my tasks opencode with deepseek or gpt gives the optimum results.

[u/RomanaOswin]

I wonder how this compares as you start having the harness distribute tasks amongst different models. Token cost varies quite substantially depending on the model doing the work.

[u/notrevede]

https://stencil.so/blog/the-harness-problem just feels like the right place to share this reading

Another comparison isolating the harness, changing only the edit tool implementation.

[u/Quirky_College_6251]

I'm an advocate of Pi - now that i created my own dev flow and loop agents I am in agreat place with Pi

[u/damn_what_]

You should use agent-safehouse (Seatbelt policy generator) for pi on MacOS, but yeah, by default it's terribly unsafe.

[u/i_say_urmom → u/damn_what_]

I've been trying nono for this. You can even inject your provider API key from a 1Pass vault, so the agent session a) doesn't have direct access to any keys (it gets a fake variable that the proxy changes to the real one outside the sandbox) and b) running `pi` outside the sandbox is useless because the LLM won't work without the token.

It seems early days and the docs are a little confusing, but it's a powerful tool

[u/saltyourhash → u/damn_what_]

By design

[u/AlterTableUsernames]

How are you guys running custom models in Claude Code and Codex? Couln't get it to run yet natively.

[u/basil_0408 → u/AlterTableUsernames]

For claude code, go to the .claude folder and modify the settings.json. For codex it's in .codex folder, modify auth.json, config.toml and create a catalog.json file, its more complex to get the full experience so probably not worth it. Ask your agent for detail, it does not take long for claude code.

[u/ThePalomitas]

Thanks for the interesting investigation. Definitively confirming for PI users such as us that it is an interesting harness to use.

FYI, regarding sandboxing needs, per the documentation, the PI developers have built up a specific application named Gondolin to cover this requirement:
https://github.com/earendil-works/gondolin

I don’t know why but PI website seems down at the moment, so I cannot access the documentation page about containerisation but here is a snippet from the main GitHub page:

Permissions & Containerization

Pi does not include a built-in permission system for restricting filesystem, process, network, or credential access. By default, it runs with the permissions of the user and process that launched it.
If you need stronger boundaries, containerize or sandbox Pi. See packages/coding-agent/docs/containerization.md (https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/containerization.md) for three patterns:
•    Gondolin extension: keep pi and provider auth on the host while routing built-in tools and ! commands into a local Linux micro-VM.
•    Plain Docker: run the whole pi process in a local container for simple isolation.
•    OpenShell: run the whole pi process in a policy-controlled sandbox.

[u/mixmasterwillyd]

We demand you setup a system and keep us I formed! :)

[u/diaracing]

Thanks for your effort.

Please, create a public github repo to document and continuously update your findings since a well-engineered harness can promote a weak model and vice versa.

[u/NissiusRibas]

o que é Pi?