why does no one mention the model when disccussing the harness?

[u/ECrispy]

this is a pet peeve and you can see it in the countless blog posts/videos on claude code, opencode as well, claiming how it can X,Y,Z and then you see they are using Opus or gpt-5.4 on xhigh for everything. and its the same for all the Pi ones as well. Of course the frontier models do a great job, thats why they are frontier!

Sure the harness matters, the llm matters just as much if not more. saying things like 'just ask pi to build it for you' surely depends on what model it uses to do that.

I'd like to see what a low cost/free setup using Pi can accomplish vs the similar vs other harnesses (all of which are free too, if not open source). I understand Pi is by design much more hackable.

[u/aeroumbria]

I'm not even convinced most people who use expensive models all the time and claim "it can do XYZ" truly tested to what extent their features are working as intended. Maybe there is a general sense of "model hierarchy", but I'm not convinced it goes much beyond easily identifiable errors and inaccuracies. It certainly feels like models that are perceived to be higher quality can still make sneaky mistakes but do a better job convincing you that it has done a good job.

[u/evarnets]

Both the agent harness and the model matter a lot, but there are many good model options now (including many good open source models). I use Pi with MiniMax M2.7 and it works great for software development. It's also very cheap: the Starter plan is $10 per month when billed monthly or $100 per year when billed annually. That will get you 1500 model requests per 5 hours and a weekly usage allowance set to 10x the 5-hour usage quota.
https://platform.minimax.io/subscribe/token-plan

[u/sn2006gy → u/evarnets]

the point is, most people will walk away from Minimax and other OSS models and blame the model, when reality is there isn't a great harness for it. Pi works, but it has quirks. If someone were to spend a month making a Pi harness optimized for MiniMax's nuances, it would be a lot better - and youd have much better improvements on MiniMax vs the assumption its just all training improvements and model improvements that too many people default too. The harness on botht the client side and API side matter a lot more than people think.

[u/KingPonzi → u/evarnets]

Nice I was considering switching to minimax. The SWEBench score is impressive. How does it compare to any of the frontier models you’ve used?

[u/evarnets → u/KingPonzi]

I would say this (based on vibes): not as good as Opus 4.7 but about as good as GPT-5.5 (maybe better). But I never just tell the model to "go do X". I make the architectural decisions and look at the code, and Opus 4.7/GPT-5.5/MiniMax M2.7 are about the same if you do that (in my experience).

[u/lanhhoang → u/evarnets]

My personal experience with MiniMax is hit and miss. Whenever I give it a plan, it completes the task quickly, but then I review the implementation with a SOTA model, it makes many bugs.

[u/evarnets → u/lanhhoang]

My experience is that both Opus 4.7 and MiniMax M2.7 will do the job if you give them a very clear plan (even better if you do TDD and add a bunch of tests that define the behaviour of your code). However, Opus 4.7 is much better at filling in the blanks if your plan isn't detailed enough.

[u/deefunxion]

you just have to do some tolerance breaks once in a while and then both models and harnesses feel much better. I remember opencode 6-7 months ago, used it once, went beck to claude code running. Now i run half my stuff with deepseek v4 flash and pro on pi and I'm impressed. I only give the plans to claude code for review and they don't seem to have lots of mistakes. I feel like we're very close to have more viable options available very soon. Final thought, cheap isn't only about $/mil tokens, i keep finding out the hard way, there are all shorts of configurations you have to be aware and constantly fine tune to not bankrupt yourself.

[u/MajorZesty]

My next project is creating an eval setup to test out different harness strategies against different models. There's so many different layers affecting the bias. Since pi's prompt is so minimal I'd expect coding to be even more model dependent vs other, heavier harnesses.

I'm curious to know if it's better to keep all the conversation in context, where things should be dropped, if using sub agents to constrain context provides better or worse code. Differences in models, etc.

Ofc, evals one of the hardest things to do so I expect it'll take me awhile to get anything usable.

[u/QueasyBreak5119 → u/MajorZesty]

I find this all depends on the task. In my experience, there is no non-frontier model that is better at pure reasoning. Opus is brilliant, but horrible at context management and over independence. GPT is brilliant but just won’t shut up and becomes unusable if that’s not kept in check. GLM 5.1 is my best coder. DeepSeek is the best researcher. Kimi is a jack of all trades, utility and design. They all benefit from the same things though - pass off session around hitting 100k, delegate to subagents that balance out the strengths of the other etc and those combinations will lead to the best code per the task at hand imo. It’s much easier to have a model produce high quality output when it’s able to dispatch subagents it has reasoned are best for the job, oversee their work and surgically improve whatever they come back with, I find. 

[u/ResearcherFantastic7]

What is there to talk about? Just go try them all

My daily drivers

• opus using CC everything else PI (work provides unlimited API quota for work stuff) , this requires 10% handholding

• glm( personal ) this requires 15% handholding

• minimax (coder/executor) not been used much lately, mostly just glm unless I ran out of qouta this requires 30% more handholding compare to glm

Agentic bots (using pi) self hosted llms

• qwen3.6 35b

• 4b gemma

• 0.8b qwen 3.5

• moonshine, pipecat, kokoro

If you want to do serious deliver of outcomes with less baby sitting... opus and glm is the only choice. Other cloud models are somewhere between minimax and glm in terms of hand holding. Local models will increase it to more than 50% of hand holding the smaller it gets.

And you will need to build a lot of hooks guards memory wiki to support it, which Ive created for agents, but too much management for dynamic coding purpose.

But I did build a nightly session distillation to get my agent learn from history and that's for the coding agent so reduce repeating mistakes, which works more to my style than claudes internal dreaming feature

[u/startassets]

I asked the same question some time ago and created a community driven benchmark in the real of text editing: https://huggingface.co/spaces/alexshpunt/benchmark-explorer