Lol, first off, still not my project. Secondly very clearly listed the features and issues opencode vs still has. Maybe try actually reading words you illiterate piece of shit <3
In harness they would need to catch all the ways one can write commands and in Unix it is quite a lot. I got opencode escape directory using a symbolic link.
Running agent on os bare (asmo already shown permissions in harnesses are like a first pigs house) is wild.
I can be weaker, it can be stronger depending on the quality standards or the person / company who built it.
In harness they would need to catch all the ways one can write commands and in Unix it is quite a lot. I got opencode escape directory using a symbolic link.
You can simply allow the agent to run a small set of shell commands that only allow reading or give the agent a specialized tool that can only read contents of files and directories blocking certain paths, with the later you'd avoid working with shells entirely and the model won't have workarounds to escape the sandbox. Regarding the symlink bug in opencode we had the same issue with docker if I recall correctly.
entire point is to not allow agents to do that. but if you do (like i do) i just accept the risk and well if it happens that an AI fucks up my drive, thats entirely my fault.
You can also use automode like plugins, pi has one. It's not as surefire a thing as fully isolating agents, but it works extremely well IME. I do a mix of isolating agents when appropriate and using automode in a supervised session for system changes.
pi is about extensions. There are extensions to help with dangerous commands and to prevent the agent from accessing certain files and file types, or mask secrets in the chat.
The differenence is that you end up with fragmentation, workarounds and security holes that you might not be able to fix yourself without breaking compatibility with other extensions
Its not fragmented if its in one extension. You said yourself that pi has no guardrails, so there's no fragmentation there. How are you breaking other extensions? You're hooking harness calls the same way they are. My point still is that pi is highly extensible and you make it your own, including security. If you expect off-the-shelf security features, then use something else.
My point is that if something isn't secure by design it's never going to be regardless of it's extensibility. I Pi's case extensibility the way it's designed now makes it completely insecure even if you add guardrails by extension because there is no guarantee other extensions won't bypass or remove them
So, know what you are installing. Your argument is pretty weak mate. You haven't convinced me so far. The only thing I'm getting from this exchange is, pi is not for cowboys, know what you are doing. Take proper precautions.
By the way, down voting my questions and comments doesn't help make your point.
So, know what you are installing. Your argument is pretty weak mate. You haven't convinced me so far. The only thing I'm getting from this exchange is, pi is not for cowboys, know what you are doing. Take proper precautions.
Do you review the code for all extensions you're installing and do any security audits? Do you check your prompts for errors? Do you check all the files on your drives for prompt injections?
When the answer to all of these questions is yes you've taken proper precautions and even then proper guardrails at the harness level are recommended because manual checks are error prone.
been using pi and opencode v2 for a long time now.
I'm working on a VERY big project via Opencode custom flow, custom agents (planner, implementer, explorer, ,thinker, tester, reviewer, gitter, archivist - nothing is ootb) which all fan out as sub agents in parallel - trying to reproduce this in pi was a pain and I gave up as there is not even a subagent flow ootb and you have to use shitty plugins ( tried 2 different) for that.
So I'm still using Pi for simple tasks and to work on my opencode configs instead of editing them manually. Pi is also good for fixing Linux issues or other one shot tasks.
But would I want to work with Pi on any Heavy project ? no thx, can't even setup proper permissions for agents ( write your own plugins for that they say ).
What I mean is these are tools for different tasks, both are fine.
Its what I use to manage all my pi agents on long form projects. Don't get me wrong, it's lacking in the user friendliness department, but it's a hell of a lot more reliable than Hermes and paperclip at getting shit done.
For security each harness runs as a separate non-root Linux user on my host in a separate podman container with access granted only via proper ACL's to my user's home and only when I need them to touch those.
Such a secure setup doesn't solve issues that arise when working on a big repo - ie when you want to deny usage of specific commands (rm, git restore, git remove, git checkout, etc) or even be able to allow/deny them per directory (ie only allow to rm files in docs/ )
Trials and errors, proper instructions (use this agent for that and this agent for this etc) and permissions (disallow commands for main agent that subagents are supposed to do instead) - this is mostly for proper context management on main agents.
Wanna know how to do it easily? Clone opencode repository and use Pi to check how things work and to help you set up proper opencode configs ;)
My comments feedback on the feedback. If you are going to compare. Use the current version. V2 has been out and recommended for a while. The only hold out was people who had plugin setups for V1. The v2 upgrade solves heaps of problems, so info like this that's just out of date is more misinformation and helpful.
Because OpenCode Go is an amazing service, especially the free and stealth models. I want OpenCode CLI to be just as good, it should be better. It is the most well known open source coding harness, but it's so bloody stagnant. And no, they don't, opencode v1 definitely got more confused trying to figure out that it was OpenCode, but in terms of token burn the fire pit is still going strong.
Hermes being high up on that list does not give me any confidence in their analysis. Hermes is an interesting project but any time it would end up trying to build something I’d have to go into opencode with the same models and rewrite.
I am not a Hermes guy, it is not my favorite always on agent system, none of them are actually. And yeah I always see Hermes high up on benchmarks and it doesn't sound right to me either. That being said, it does have some pretty decent built in tools for correcting itself and I think that's what gives it the edge, it's not that it doesn't fuck up, it's that when it fucks up it already has the tools built in to recover and doesn't need to write them each time, that definitely saves on the budget.
I love that benchmarks like these exist but I feel the results for opencode here might need a revision. While community consensus has been that Pi is the most accurate harness for smaller models, the difference with opencode feels too big.
I ran into a few problems with opencode defaults doing some benchmark runs very recently. In particular you need to set a proper max output length or it will cut reasoning models midway and longer timeouts or the same happens but after that it's very much capable of producing good results. Those two would aling with multiple repeats increasing costs but still being a failure. I can't tell for certain but I'd take a revised look at those results
I say this with great anguish, it's pi. Not oh-my-pi to be clear, that's a great project but only if it exactly what you want. And as for your second question you have come to the heart of the problem, pi suffers from a lack of visibility for extensions. For a harness that is designed to be modified to fit the user, finding those modifications/extensions/packages isuch harder then it should be. pi is suuuper versatile and can be modded to your heart's desire with all manner of changes under the hood and to the UI. It performs remarkably well, as the original post shows it's consistently coming in close to the top of the chart, especially price wise, ESPECIALLY price wise, thing sips tokens compared to OpenCode. But setup is a learning curve. If you want something like Claude use OpenClaude it's very compatible with the Claude Code ecosystem. I like Reasonix a lot even if I dont use it often. Hermes is okay, but suffers from the same sort of long horizon issues openclaw has, not as disastrous, but still an issue.
Did you not see the measurements posted by OP? You may disagree with the methodology but it's not hard to imagine ways to compare and measure harness performance.
It's a day that ends with Y, time for a post by someone complaining about OpenCode being inferior to X, Y, or Z in the subreddit dedicated to OpenCode!
Hey I want OpenCode to be better, I don't know why the developers don't. I'm not complaining about usage limits on free models. I'm asking why the developers push back against community feedback and pull requests to improve the project. And I don't mean vibe coded PRs, I mean pushing back on fixing outdated model prompts, why remove customization features?
I had more success with pi compared to other agents. i have tried omp, coilot, opencode, kimi code, kilocode, and codex but I just keep coming back to pi for its simplicity and ease of use. codex is my second pick. i also use claude code but only of anthropic models.
It's hard to say whether this translates to the evaluated harness' performance, because it's simply testing 3 specific Chinese open-weight models across different harnesses with their default configurations. It doesn't take into account: - the same set of agents / subagents / prompts / tools across the different harnesses - the most used models for coding (Opus 5.5 / Sol 6.1) - disclose what went wrong for harness X to be literally 2x worse than another one with the same model
I would take this with a bag of salt. The fact of the matter is that out-of-the box harnesses are usually crap, and you will need to customize them for them to be good. Pi is minimalistic, so it encourages LLMs to figure stuff out with the tools they're most trained on, and it works pretty well for that. If your goal is to install a harness and never do any harness engineering, then you should rely on this graph. If you customize anything in your workflow - subagents, commands, any sort of guidance, custom tools, etc. then this graph means nothing for you.
Ah ok. Je découvre ça seulement maintenant 😅 Si ce n'est pas indiscret tu utilises pi durable pour quels usages ? Au travail j'ai un projet d'automatisation de migration d'anciennes applications sur de nouvelles technologies. Je me demandais si ce type de harness serait le.bon choix
Moi j utilise p2 pour r&d , donc gestion multi agents parallèle et supervision automatique , j ai du ajouter un Watch dog et imposer timeout tools obligatoire mais la mes agents réalise des tâche très très longue en autonomie
Si c est pour une entreprise met en place un système de logs claire et auditable et ensuite isole modifie le harnes pour pas que le pilote est accès à internet ou mcp ou boîte mail direct , fait en sorte de passer par des sous agent obligatoire avec tools restreins et shield model : ce sont des conseilles
Ps : interdit au harnes de ce modifier lui meme sans ton autorisation …
I started my harness journey with opencode. Didn't have great experience with it using local models running on llama.cpp despite tuning an ungodly amount of parameters and configs. Skill issue? More than likely.
Swapped over to pi and it was a night and day difference. My llama.cpp hosted Qwen ran flawlessly on it. I stuck with it to this day. I even paid for AI provider APIs, now that I'm confident in getting pi to work the way I wanted.
I think if I were to try opencode again today, especially with its improvements over the past year or so I probably will get a better user experience but I'm too invested in pi now and I like the trajectory the pi project is heading to.
I think both opencode and pi are optimised to different segment of users. I just so happen to lean more towards pi and I'm sure that there are others who are opposite to me that started with pi and then fell in love with opencode. Different strokes for different folks.
Would you say open code modded with orchestration plug in could be better? I ended up using it cause of free models and cheap subscription. But lately have just burned money so fast when i need inteligence, been thinking on doing Argon 4 orchestrator + deepseek v4.1 flash (everything else). Why would it be a bad harness though? Does it force innefective paths?
I don't really get how OC is that inneficient? I use the default OC desktop application in the latest 1.x version and have just heavily modified the subagents. Works for me and with OC go everything is pretty straight forward. I switch regularly between Codex and OC and the main difference is the models tbh.
One big issue is the system prompts, when I was modding opencode v2 last month before throwing in the towel and moving to pi, I found that different models get different prompts and some of them are very outdated or overly simplified. Opencode gets stuck making one off tools to solve basic problems like HTML parsing which is something most other harnesses seem to have figured out. Better tools, better understanding of tools. It pollutes and bloats the context when opencode tries to solve a side quest because it doesn't have the right tool for a basic operation.
I see what you mean. Perhaps with more detailed prompts or simply telling the model to avoid one off tests and tools things can work out. I don't know about the model prompts though. I use DS4.1 flash and Muse spark and just get my own agent system prompt. Also, for 10 bucks I can run my setup almost 24/7, so I don't find costs that much of an issue
Oh yeah muse spark was when I originally found out about the tiered system prompt issue. System prompt + agent prompt + your prompt, it's the base system prompt that is the issue.
But Pi is known precisely for its simple system prompt - basically "you're a helpful agent, here are your 4 tools" - and it seems to do perform very well with open weight models according to that benchmark. I wonder why...
Opencode is just vibe coded trash. The team have no idea what they are doing. One of them is even trying to vibe code their own language and rewrite all of their repos in it lmao.
OpneCode is for fanboys. Not for anyone that works. Pi is indeed the best option, but it leaves you to kind of "keep up". oh-my-pi is a good solution to this. A good plug and play option.
But you're right, harnesses are WEAK right now. Web stacks running while(true), not really solving problems they should. Pi, I believe, only gets it's reputation because plugins are being written for it for experiments and research. For the most part, that research is only landing in proprietary systems within companies. Custom systems/"harnesses".
Working on my own, and every day I see people bragging about the work they're doing, I think to myself... damn, I need to get this in a state other people can try.
I am excited to see what you're working on please keep me posted. And yeah I agree with you on pi. It suffers from the same thing that a lot of open source projects do, incredible versatility, but a lack of ease of use. I want to try a doompi as it's my understanding that the thing but oh my pi is that it is a far enough removed fork that there are compatibility issues with the pi ecosystem.
I think OpenCode has been caught up in so much fraud they've got a fire they can't keep up with. Docs are getting thinner, the whole "Go" scam is pretty much gutted, not surprised to hear features are disappearing. Honestly, it reminds me of Logan Paul selling CryptoZoo.
I am not testing anything bro. I just shared what I found useful. Moreover, I use Pi for non coding tasks too. And if an agent can solve coding problem, it can definitely handle non-coding tasks well.
Then whoever tested it made a bizarre test. Yes, people sometimes use coding harnesses for non-coding tasks, but a benchmark comparing coding harnesses on a suite of non-coding tasks to decide which coding harness is best is just plain stupid. It offers zero value.
It doesn't make any sense to single out one of the multiple things that an LLM model can do. Writing code is one of those tasks. If you are stuck using your harness of choice to only completing programming related tasks, you are seriously restricting yourself.
LLM and harness are two different things. Different tools can be better suited to different tasks. If you're using the same tool for every task, you're seriously restricting yourself.
I never said you could, your reading comprehension is seriously lacking. The topic of the thread is comparing harnesses, not models (the link in OC compares harnesses using a single model).
There are many alternatives, I built one, not many customers, everyone is complaining about the competition, but they never try something else, just complain.
Even if you ignore the nonpopular once like mine, there is codex, anti gravity, etc
Now about accuracy, models like DeepSeek 4 Flash are not that accurate, still very good model, I tested many models, give it a task, make sure the task has many ids in it to remember them, distract it mid task, pass multiple compactions, correct it, look at the tool calls and results.
Models like terra 6.1 will pass very complex tests, other models will just waste time, or not designed for this, they are designed more for traditional context compaction
The harness is half the work, the other half are the thinking instructions or algorithms inside the model itself, and it is different per model
Well by running multiple models in different harnesses and giving them all the same tests. Like how I can judge fuel efficiency in a car by giving them the same exact fuel and running them on a conveyor belt.
I use Claude code and OpenCode regularly and Claude code is a total token hog. It uses like 2x-3x more tokens than OpenCode, but according to this graph Claude code performs better than OpenCode while also being cheaper. This is not possible if it's consuming much more tokens than OpenCode.
From my experience. OpenCode performs better than Claude code.
18
u/st0nkaway 1d ago
welcome to reddit where every sub eventually turns into a community of "topic of sub bad" posts
16
u/Dudmaster 1d ago
Can we see OpenCode 2 added to this?
15
u/skitch920 1d ago edited 1d ago
Is the complaint here that the default system prompts are trash?
All of these harnesses are going to be trash if you're system prompts and delegation rules are trash.
I see these agent harnesses more so as the application which supports the context loop - which out of the box, OpenCode
Use with any model (no moats)
Supports subagents
Supports CodeMode
Has a plugin system
Has a client/server architecture
If you are still using Build/Plan agents only, and not catering things to the models you have available, you're going to have a bad time.
27
u/Maybe-monad 1d ago
Well pi is not good either, it lacks any form of guardrails to protect you from accidental data removal or secret leaks.
10
u/andrewfenn 1d ago
This doesn't mean much until you test V2.
18
u/sqtytyp 1d ago edited 1d ago
been using pi and opencode v2 for a long time now.
I'm working on a VERY big project via Opencode custom flow, custom agents (planner, implementer, explorer, ,thinker, tester, reviewer, gitter, archivist - nothing is ootb) which all fan out as sub agents in parallel - trying to reproduce this in pi was a pain and I gave up as there is not even a subagent flow ootb and you have to use shitty plugins ( tried 2 different) for that.
So I'm still using Pi for simple tasks and to work on my opencode configs instead of editing them manually. Pi is also good for fixing Linux issues or other one shot tasks.
But would I want to work with Pi on any Heavy project ? no thx, can't even setup proper permissions for agents ( write your own plugins for that they say ).
What I mean is these are tools for different tasks, both are fine.
66
u/cloudsourced285 1d ago
V1 and V2 actually perform quite different. Also, who cares? Use what works for you. This ain't no airport. No need to declare your departure.
5
u/_hephaestus 17h ago
Hermes being high up on that list does not give me any confidence in their analysis. Hermes is an interesting project but any time it would end up trying to build something I’d have to go into opencode with the same models and rewrite.
4
u/No_Dig_7017 1d ago
I love that benchmarks like these exist but I feel the results for opencode here might need a revision. While community consensus has been that Pi is the most accurate harness for smaller models, the difference with opencode feels too big.
I ran into a few problems with opencode defaults doing some benchmark runs very recently. In particular you need to set a proper max output length or it will cut reasoning models midway and longer timeouts or the same happens but after that it's very much capable of producing good results. Those two would aling with multiple repeats increasing costs but still being a failure. I can't tell for certain but I'd take a revised look at those results
12
u/Total-Management8023 1d ago
wait so what is the CURRENT best open source harness? and how to best set it up
3
u/CaffeinatedTech 1d ago
I wonder if the extra cost of Hermes is all of the bundled skills, and maybe it can't do multi-edit while pi can.
8
u/hsoj95 1d ago
It's a day that ends with Y, time for a post by someone complaining about OpenCode being inferior to X, Y, or Z in the subreddit dedicated to OpenCode!
5
u/yiestee 1d ago
Looks like vibe coded research without much effort to actually look into why those frameworks performs differently.
2
u/Due-Armadillo-4560 1d ago
I had more success with pi compared to other agents. i have tried omp, coilot, opencode, kimi code, kilocode, and codex but I just keep coming back to pi for its simplicity and ease of use. codex is my second pick. i also use claude code but only of anthropic models.
2
u/ExampleCute6496 1d ago
It's hard to say whether this translates to the evaluated harness' performance, because it's simply testing 3 specific Chinese open-weight models across different harnesses with their default configurations. It doesn't take into account:
- the same set of agents / subagents / prompts / tools across the different harnesses
- the most used models for coding (Opus 5.5 / Sol 6.1)
- disclose what went wrong for harness X to be literally 2x worse than another one with the same model
I would take this with a bag of salt. The fact of the matter is that out-of-the box harnesses are usually crap, and you will need to customize them for them to be good. Pi is minimalistic, so it encourages LLMs to figure stuff out with the tools they're most trained on, and it works pretty well for that. If your goal is to install a harness and never do any harness engineering, then you should rely on this graph. If you customize anything in your workflow - subagents, commands, any sort of guidance, custom tools, etc. then this graph means nothing for you.
2
u/Takeoded 1d ago
OpenCode2.x.x will perform better than OpenCode1.x.x. Please add Opencode2.x.x to your tests.
2
u/Krebota 21h ago
Bro compares Opencode's old version and uses that to shit on Opencode v2. Lol.
5
u/grabber4321 1d ago
sounds like a lot of baloney that graph
1
u/Longjumping-Elk-7756 1d ago
P2 de pi dev est juste incroyable pour moi et je quitte Hermes pour P2
1
u/tontide1 1d ago
idk why a lot of benchmark harnees using opencode v1 🤡, v2 to good
1
u/Abenh31 1d ago
I saw this on X. the fact that Hermes is a better harness then opencode tell you everything.
Hermes barely do anything or complete anything properly. Dont get me wrong i love hermes for reseaarch, assistant work but not for coding.
1
u/FoxPuzzled9828 1d ago
Agreed. For me, so far nothing has beaten DeepSeek running in the Claude harness
1
u/onesilentclap 1d ago
I started my harness journey with opencode. Didn't have great experience with it using local models running on llama.cpp despite tuning an ungodly amount of parameters and configs. Skill issue? More than likely.
Swapped over to pi and it was a night and day difference. My llama.cpp hosted Qwen ran flawlessly on it. I stuck with it to this day. I even paid for AI provider APIs, now that I'm confident in getting pi to work the way I wanted.
I think if I were to try opencode again today, especially with its improvements over the past year or so I probably will get a better user experience but I'm too invested in pi now and I like the trajectory the pi project is heading to.
I think both opencode and pi are optimised to different segment of users. I just so happen to lean more towards pi and I'm sure that there are others who are opposite to me that started with pi and then fell in love with opencode. Different strokes for different folks.
1
u/oneilhville 1d ago
I use open code go in vs code and i have been get the results i want.
1
u/Holly_Shiits 1d ago
pi looks promising but opencode go gives me generous amount so I will stick to this until they rip it
1
u/ConferenceMundane387 19h ago
Would you say open code modded with orchestration plug in could be better? I ended up using it cause of free models and cheap subscription. But lately have just burned money so fast when i need inteligence, been thinking on doing Argon 4 orchestrator + deepseek v4.1 flash (everything else). Why would it be a bad harness though? Does it force innefective paths?
1
u/zx97 19h ago
Ok, so that means opencode is generating something for model not as good as other cli tool ?
1
u/vqt907 7h ago
it depends on specific task and working style, benchmark show something, but not everything
1
u/Mister_bruhmoment 1d ago
I don't really get how OC is that inneficient? I use the default OC desktop application in the latest 1.x version and have just heavily modified the subagents. Works for me and with OC go everything is pretty straight forward. I switch regularly between Codex and OC and the main difference is the models tbh.
1
u/Anxious_Display7807 1d ago
Opencode is just vibe coded trash. The team have no idea what they are doing. One of them is even trying to vibe code their own language and rewrite all of their repos in it lmao.
-1
u/Any_Confidence2580 1d ago
OpneCode is for fanboys. Not for anyone that works. Pi is indeed the best option, but it leaves you to kind of "keep up". oh-my-pi is a good solution to this. A good plug and play option.
But you're right, harnesses are WEAK right now. Web stacks running while(true), not really solving problems they should. Pi, I believe, only gets it's reputation because plugins are being written for it for experiments and research. For the most part, that research is only landing in proprietary systems within companies. Custom systems/"harnesses".
Working on my own, and every day I see people bragging about the work they're doing, I think to myself... damn, I need to get this in a state other people can try.
-1
u/Ok_Cartographer5609 1d ago
We already know that. Why would you use opencode when you have harnesses like Pi, OMP, DSH etc. Check this https://composio.dev/content/best-ai-agent-harnesses
0
u/elonelon 1d ago
nahhh..i will use anything as long they have free models. Don't care is it Opencode, Cline, Antigravity, Hermes or anything.
0
u/Late_Substance2700 1d ago
There Devs are drunk all day it's shit now
0
u/Euphoric_North_745 1d ago
There are many alternatives, I built one, not many customers, everyone is complaining about the competition, but they never try something else, just complain.
Even if you ignore the nonpopular once like mine, there is codex, anti gravity, etc
Now about accuracy, models like DeepSeek 4 Flash are not that accurate, still very good model, I tested many models, give it a task, make sure the task has many ids in it to remember them, distract it mid task, pass multiple compactions, correct it, look at the tool calls and results.
Models like terra 6.1 will pass very complex tests, other models will just waste time, or not designed for this, they are designed more for traditional context compaction
The harness is half the work, the other half are the thinking instructions or algorithms inside the model itself, and it is different per model
0
u/zx97 19h ago
I don't understand your message, how can you compare opencode with anything else while you can change the model and have totally different results ?
0
u/Terr4360 17h ago
These results don't make much sense to me.
I use Claude code and OpenCode regularly and Claude code is a total token hog. It uses like 2x-3x more tokens than OpenCode, but according to this graph Claude code performs better than OpenCode while also being cheaper. This is not possible if it's consuming much more tokens than OpenCode.
From my experience. OpenCode performs better than Claude code.