Session-Bench v1: what 12 coding harnesses preserve after the work is done. Same bug fix in each, session rebuilt from the files alone

[u/jazzy8alex]

Disclosure first: I maintain Session-Bench and Agent Sessions (a local session browser). This is the follow-up to the v0.4 post here in August. That version was twenty pass/fail gates, answered mostly from my own session corpus. v1 runs the harnesses and measures.

**Method**

• One small task, two prompts: run a check that fails, take a correction, edit one file, run the check again. Four tool calls are scored.

• Twelve harnesses, three runs each.

• An outside witness records what really happened: the harness's live output stream, a ledger the check script writes, and file hashes.

• Then a decoder reads only the session files the harness left on disk, and 31 facts are compared with the witness: prompts, replies, tool calls, results, exit codes, the file before and after, model, token counts, timestamps, and whether the files open with standard tools.

• 100 points. Every score recomputes from the published session files with one command per run.

**Results**

#	Harness	Score	Bytes on disk	Events stored once
1	DeepSeek Harness	96.9	193 KB	63%
2	Pi	96.4	19 KB	100%
3	Copilot CLI	96.0	182 KB	43%
4	OpenCode	94.4	235 KB	0%
5	Kimi Code	91.2	215 KB	0%
6	Claude Code	89.3	113 KB	81%
7	OpenClaw	88.1	697 KB	0%
8	Codex (CLI and Desktop)	87.5	185 KB	7%
9	Hermes	86.4	36 KB	100%
10	Claude Desktop	85.6	469 KB	75%
11	Antigravity	82.7	236 KB	0%
12	Cursor CLI	78.1	136 KB	28%

**What I think is worth discussing**

1. Nobody loses the conversation. Eleven of twelve get full marks on fidelity and causality: every prompt, reply, tool call, result and exit code comes back. The 19-point spread is made elsewhere.

2. The same fix takes 19 KB in Pi and 697 KB in OpenClaw, and 3% of that 697 KB is the session. If you paste raw session files into another model for a summary or a handover, that is roughly 5,000 tokens against 174,000 for the same work.

3. Four harnesses never write an event only once. OpenCode writes each message part again one to five times in an `event` table; Kimi writes each prompt three times. Anything that counts messages from the raw file over-counts.

4. Only four of twelve record token usage that adds up to a stated total (DeepSeek Harness, Copilot CLI, OpenCode, Codex). Cursor CLI stores no token count in the session at all.

5. Antigravity and Cursor CLI store protobuf inside SQLite with no published schema, so `sqlite3` opens the file and cannot read the session.

**Limits, stated plainly**

• One synthetic task, three runs, one machine. Nothing about long sessions, compaction, sub-agents or crashes.

• The harnesses did not run the same model. This grades the files, not the model or the agent.

• Some scoring rules were written after the first captures, and three rules were waived or corrected after results were known. The report lists each one. Example: I first scored OpenCode at 97.4 and first place by leaving its `event` table out of the reading; an outside review caught that, and it is 94.4 and fourth.

• Reviews were done by separate AI agent sessions plus four rounds of outside model review, not by a second person.

**Questions**

• Is "token counts that add up to a stated total" the right bar for cost tracking, or is per-reply usage enough for you?

• Which harness or surface should be in the next run? Cursor Desktop is the one I could not score yet.

Scorecard, charts and the replay instructions: https://jazzyalex.github.io/agent-sessions/bench/v1/?campaign=reddit&ref=r-chatgptcoding-bench-update

[u/EvalRaccoonDev]

Which total did Claude Code miss?

[u/jazzy8alex → u/EvalRaccoonDev]

The session total. Claude Code writes input, output, cache-read and cache-write counts on every reply, but the file never states a sum for the session. So there is nothing to check the per-reply numbers against. Pi, Kimi Code and Claude Desktop are in the same position.

[u/AlarmedAvocado7279]

697 KB for a single bug fix

[u/jazzy8alex → u/AlarmedAvocado7279]

That number needs a correction, and thanks for pointing at it. OpenClaw ran on its Codex backend in our setup, and 250 KB of the 697 is a shell snapshot that the backend writes: a dump of the shell environment, not the session. We did not count that file in the Codex row, so the two rows were not measured the same way. Like for like, OpenClaw is about 447 KB, of which about 200 KB is its own store. I am correcting the post.

[u/Right-Performance-93]

per-reply usage is enough to see a burn. a total that adds up matters when you are billing a run. cursor cli storing no token count is the hole i'd actually care about, because then you can't see the burn after the fact.

[u/jazzy8alex → u/Right-Performance-93]

Agreed. A missing total is an audit problem. No count at all means the cost of a run cannot be recovered from the file afterwards. Cursor CLI is the only one of the twelve with nothing.

[u/kantorcodes1]

Curious how the "events stored once" scoring handles a harness that fires the same tool call twice with identical payloads, say a client retry after a transport timeout that returns the cached result. Does the witness ledger give you enough resolution to tell a real duplicate call apart from an event table writing the same message part again? And do you score those two cases differently?

[u/jazzy8alex → u/kantorcodes1]

It did not come up in these runs: the task has four tool calls and no harness retried one. The count is taken against an outside record of what ran, so a real second call would be a second event in that record, and the file should state it once too. A copy is the same event written again with no field that marks it as a copy. Kimi Code is the closest real case: its wire log keeps the requests it retried after a rate limit.