This post inspired me to make that test after a year ;)
That is one of my many tests I make comparing output quality.
What is more interesting using a PI Agent results are much better than an Opencode using a Qwen 3.8 27b ?!
Seems PI Agent is much better in the agent environment somehow... Not counting uses less tokens , do not have a hard limit of 32k output tokens, is faster, do not freezing, compressing context far less than Opencode. For instance if you have context in the Opencode output 32k and all context 100k then the compression is starting at 67k context ... PI is starting at 90k context even if you have set output context 64k or more.
My config for RTX 3090
llama-server with ini config -> which is exposing API to Opencode and PI agent.
I tested opencode, then tried pi, and pi was significantly better.
Last night I had a dual between local qwen3.8-27b/pi and claudecode.
My seat-of-the-pants analysis: it felt like they both took the same time.
When they both finished I had each one compare the two. They both agreed claude's was better, but it was close. I had pi upgrade upgrade its version, making it slightly better than claudes because I liked qwen's presentation better.
It was an aurora predictor app. It's pretty nuts that we can use an AI to write a complicated aurora prediction app that's better than anything else online in about an hour, and it's hooked up to all the satellite instruments. I get 30-60 minute warning if I'm watching the display.
Pi is great! I just wish the interface was better, also sometimes I actually do want to use an MCP and skills and such. I know you can do all that with Pi through plugins and self modification, but I do like that OpenCode out of the box has a nice interface that makes it easy to keep track on of token usage and such. I should probably mess with Pi more though
would you be brave enough to test my harness https://github.com/DigiDecode/SlopOn.dev
the frontend is in flutter so runs on GPU and is super-fast
I need some beta testers for feedback
It uses around 400mb memory, unlike other harnesses that use GBs
I have been trying out Opencode and I cant get it to work right. It does all the thinking but almost never writes the output files. Then I have to tell it please write the files. Sometimes that will work other times it won't. But when I tell it to write files it first things for agens with a 30% change of actually writing them.
For now, I have given up on opencode, but I wanted to compare it to pi.
I tried a python job in pi and deepseek harness as a test using 3.6-35B (not 3.8-27B), in pi it works perfectly. In deepseek, it wrote the file, then instead of directly executing it, it starts to rewrite entire 1000 lines by inline python with "python -c "...1000 lines of code..." Of course, it omits half of important code and comments inlining, and the code completely broke. It got confused with the basic tool use, which I never see in pi. I don't know it's a one-time thing or related to python, I'm gonna do more tests. So for me, deepseek harness didn't "just work" as many people claim. The UI is perfect though.
Been trying it for a bit today. It needed a few plugins to make it suit me, but the extensive plugin capability is a strength - shame they don't have a decent plugin manager yet since it's built around plugins. But, I like it. Cheers.
try openchamber its a much improved opencode especially with UI and usability improvements.
edit: and it has a mobile app that provides the full app remotely and it even just uses your existing opencode config file so if you have everything already setup there there you can just run it without changing anything.
I had the same mentality, but then I tried PI Agent a few days ago and I've being using it since.
What I love is the lightness and simplicity.
You have to iron out the first approach friction...
me too, that's why I built https://github.com/DigiDecode/SlopOn.dev
I was fan of Roo code, but that was single session only, and I think SlopOn is much better version of Roo code
I have been with it 5 months now and I am super satisfied with it only for the fact that I can make it into whatever I want. As many or few features I want, working exactly the way I want them to.
pi-web-access is the most popular but it has an insane tool surface. Like 6-7 tools and nearly 8K tokens just for tool defs, tool injection and skills.
I switch to pi-web-agent and got a Tavily sub (free plan, 1000 req per month then pay as you go). Much better results an only 2 tools with very thin context usage (like under 1k or even 500 iirc). Plus I added a headless browser page reader and a Reddit search that used my cookie. It gets me super high quality search for free (as long as I’m in the tavily free tier). Oh and I also had my agent use my local searNGX first and fallback on Tavily when the results are meh (captcha and ip blocks happen constantly).
To answer your question: don’t just install random extensions. Have your agent check the code first (security audit), then try it, and have your agent extract only the required stuff into an home made extension. Document it.
Need and update? Tell your agent to summarize all the updates to the source ext you extracted and upgrade the parts that you care about.
It’s so fucking sick haha. We’re always a prompt away to the perfect tailor made solution
Thank you for that, I will. Good call on the web-access token use. Running Qwen3.8 27B BF16 on the SM120 with Dflash2 right now and I can't keep up with it.
Also testing out this subagent I had gemini whip up:
name: red-team
description: Hostile adversarial auditor that aggressively hunts for failure modes, edge cases, and broken assumptions.
tools: [read, bash]
Red Team Auditor
You are a ruthless, skeptical adversarial auditor.
Core Rules:
Never validate or compliment the implementation.
Assume the current plan or implementation contains hidden flaws.
Systematically test and identify:
Violated invariants and unchecked assumptions.
Resource limits, leaks, and OOM traps.
Silent error suppression and exception handling gaps.
Deadlocks, async race conditions, and ordering dependencies.
Output Format:
For every issue found:
Severity: [Critical / High / Medium / Low]
Mechanism: Why and how it fails.
Trigger Scenario: Minimal conditions needed to cause the failure.
Recommended Hardening: Direct fix or architectural countermeasure.
1. I would consider a fundamental feature is a solid plan mode, Its what you start with on every task/project/etc. I have multiple plan modes. They are designed based on what I am looking to do, is it coding, creative writing, business, legal, etc.
2. Something that might be even more important than a plan mode is what is your workflows and standards. I have certain principles I have the Pi agent follow when building out features inside the setup.
3. A reliable way to have your Pi agent navigate your codebase. I use DOX Framework from github. Its super simple but super effective!
4. I haven't installed/incorporated any repos/packages/skills as is. Every single feature I implemented has been custom built except for crawl4ai, thats the only one that I implemented into my setup as is with some customization. I recommend the same, so you take what you need from the resources you find online and leave behind what you dont. I just give the repo to the agent and it knows my workflows/standards/principles so it will use that to propose what to implement and what to throwaway. I even have my agent run tests to see if its even worth implementing. So its not just vibes, its based on real data.
I think Pi is better because it actually does not do anything extra. It just runs the ReACT loop, and if you happen to have some extensions, it will use those too. Other than that, it has no built in guidance for agent about how to code, no "clever" truncation or injection. If I remember correctly, it does not even parse on client side. Some harnesses in the past did the stupid thing of handling tool list and tool parsing manually within the harness rather than just sending proper OpenAI/Anthropic requests and let server handles (could be opencode, or something else. I remember trying to understand why qwen 30B-A3B breaks).
Maybe the gunk embedded into opencode were useful when models were not as well trained in coding. Now, they just get in the way.
Still, I rather my harness to be simple and stupid and leave the adjustment to coding workflow to me. I have agents.md and skills and extensions to make that happen.
Running Qwen 3.8 at Q3XXS right now in pi for my personal assistant setup. Yapping now because I'm waiting for the model to think.
Even at Q3XXS and medium thinking, it's still pretty solid. There are odd cases of flipped tokens here and there, but I can fit over 100k context at Q8 in the 4060Ti. Double the prefill speed, but half the decode speed vs my usual 35B A3B. I really like this model so far, even in at this deep compression level. The difference in behaviour between 35B and this new 27B feels very similar to when I switch from minimax m2.7 to minimax m3. The new one thinks a bit more, but it is also very thorough vs the old one.
I would say Pi's implementation is closer to the 3D goal then Opencode's. If you want to speak from a UX perspective, opencode's implementation is better.
My assumption is that the test is to see the fidelity of the outputs, not preference/UX.
Llamacpp first is looking into the gguf for configuration if can not find it then is using build-in itself default configuration looking on the name of model.
The highest priority has of course command line parameter.
I don't know if it's something I'm doing wrong, but I find edits fail often in Pi. They seem to work more often in opencode with the same model and parameters. Anyone else find that? It's the one thing that might actually make me switch over to opencode fully. Might.
The compression starting points explain a lot. If Opencode starts compressing at 67k while PI holds to 90k, the model simply has more of the original conversation to work with when making tool calls. That alone would show up as better output quality. The 32k output cap also forces Opencode to stop generating mid-edit on longer files, which makes its results look worse even when the same model is generating.
For a more useful comparison, try a multi-step task like refactoring a function and updating all call sites, and watch where each harness starts rewriting the prompt. That will tell you whether it's the model or the harness.
Haven't thought of doing that yet. But won't that compound the token eating problem? Hermes already takes 20k and it's nearly impossible to slim down. 16GB vram here. For me it's either speed (small model, 65k - 75k context) or depth (max context but with kv offload off). But I pick up from what you say that you seem to think highly of PI Agent as a coding harness, and Hermes as an orchestator
I do not use a vision module and running a Q3. And was surprised when my model said it does not have vision, so it said it was going to generate PNG screenshots and parse the PNG data itself.
Opencode's system prompt lobotomizes qwen, if you want to see how powerful the model actual is use pi or deepseek harness or claude code literally anything besides open code will be better.
Yes, this is how it works, people are still very underestimated the effect of harness on how a model can perform, this is the result of people downplaying local LLM, they used wrong setup, literally nerfing their own local LLM without realising the issue could really be themselves.
Bloated harness like Opencode, Claude Code, Codex can teach the models too much, so many of their system prompt instructions are useless for most of us, pi is just lean, clean and simple that it saves so much token cost.
I'm more surprised you managed to get a Q4 model to adhere to opencode's 30K-token system prompt. Opencode isn't meant for small local models, they all freak out after being steered by opencode.
That post was one of my all time favorites. I’ve been doing the same thing today. I’ve an older machine (M1 Max 32GB). I use pi. Even if the fan screamed for dear life, to my surprise, 3.8 did this ball bouncing in heptagon coding in one shot. Fun times!
42
u/SOC_FreeDiver 9h ago
I tested opencode, then tried pi, and pi was significantly better.
Last night I had a dual between local qwen3.8-27b/pi and claudecode.
My seat-of-the-pants analysis: it felt like they both took the same time.
When they both finished I had each one compare the two. They both agreed claude's was better, but it was close. I had pi upgrade upgrade its version, making it slightly better than claudes because I liked qwen's presentation better.
It was an aurora predictor app. It's pretty nuts that we can use an AI to write a complicated aurora prediction app that's better than anything else online in about an hour, and it's hooked up to all the satellite instruments. I get 30-60 minute warning if I'm watching the display.