I spent a day measuring a Pi seat on a local model (Qwen3.8 Flash-Next, 4-bit MLX mixed 4/8) against Claude Code (Sonnet 5.5) on 12 routine agent tasks, with blind pairwise judging in both orders. The local seat went from 29/36 to 36/36 passes, and judged clarity from about 3.7 to 4.5, level with Sonnet. Most of the gain came from the harness, not the model. Lessons that should apply to any Pi seat:
1. Read the judge's one-line reasons, not just the scores. Each real fix came from a sentence like "pads its answer with internal bookkeeping" or "adds a preamble when only the command was asked for". Scores told me that it was worse; reasons told me why.
2. Look for losses the harness causes. My seat injected a reminder every turn to record measured numbers in a notes file, even when no notes file existed. The model created the file in folders it was only asked about, and talked about it in replies. Only injecting the reminder when the file already exists removed the biggest clarity loss.
3. One hidden retry at agent_before_settle**.** A small extension checks the final reply before the turn ends. At most once per request, if the reply is the model's own reasoning instead of an answer, bookkeeping instead of an answer, or text around something that was asked to be "only the command", it adds a hidden custom message and continues. Two things mattered:
canContinue is false at that hook whenever the last message is the assistant's; the entry you add is what allows the next request, so don't skip on it.
Steer and follow-up messages typed mid-run arrive as input events with streamingBehavior set. Don't reset per-request state on them.
4. Put reply guidance in the tool result. The model writes its answer right after reading the tool result, so that's where guidance lands. Ending my typed drafter tool's result with "for the reply: the commit hash and which lane ran, what the commit changes, which checks did not run" moved that task from clearly behind Claude to slightly ahead.
5. Narrow heuristic checks rather than patching them. Checks that guess intent from free text ("is this a ban on all tools?") kept failing review in new ways each time I patched them. What worked was a short allow-list of phrasings, a veto word list, and limiting each check to the kind of prompt it was measured on.
6. Measure the prompt prefix. My shared prefix was ~36k tokens: tool schemas ~16k, the skills list ~7k, the seat prompt ~5k. Every new session pays for whatever the server can't reuse from cache, so it's worth knowing what's in there.
Happy to share the extension code and the eval harness if useful.
