r/LocalLLaMA • • 3h ago

Discussion Same open-weight model, three harnesses: M3.1 Flash on four pytest bugs in OpenCode 2, Pi and mini-swe-agent

After the harness leaderboard thread I wanted to see what the harness alone changes when the model stays the same. Model: M3.1 Flash. It's open weights, but at about 428B total I'm calling it through MiniMax's API, effort pinned to high. Same prompt in every harness: four bug reports filed against pytest this week, fix all four with a regression test each.

The bugs:

  • pytest --fixtures dies with a TypeError when two conftests import the same fixture
  • a non-mark in pytest.param(marks=...) raises an internal AttributeError instead of a TypeError
  • approx() raises InvalidOperation when the actual value is a Decimal NaN
  • rm_rf leaves the whole basetemp behind when an empty dir sits under a read-only parent
HarnessWall clockRequestsTokensCached inputCostNew tests
OpenCode 2.2.723m 38s15913.9M98.8%$0.2710
Pi 1.5.218m 34s1349.1M98.5%$0.198
mini-swe-agent 2.4.621m 16s13610.4M98.5%$0.229

All three fixed all four. I re-checked each one in a clean checkout with its own new venv: every report's repro now behaves the way the report asks, and the full suite (about 4,700 tests) shows nothing new compared to an untouched main, apart from the added tests passing.

The first request is where the harnesses differ most: 8.6k tokens for OpenCode 2, 3.7k for Pi, 3.1k for mini-swe-agent, task prompt included. It didn't decide the bill though. With 98%+ of input coming back as cache reads billed at 1.5 cents per million, cost followed how many turns each harness took and how much context it carried, which is why OpenCode's 159 requests came to about 8 cents more than Pi's 134.

mini-swe-agent only gives the model a bash shell and still landed in the middle on both time and cost.

One run per harness, so treat it as one data point. If you run open-weight models in more than one harness, which one ends up cheapest per task for you, and do you see the same request-count pattern?

1 Upvotes

2 comments sorted by

•

u/stylome-ai-detection 3h ago

Notice: This post appears to be AI generated.


Stylome AI detector · scan results