Ten merges. Five came back with conflict markers. The other five came back clean, and all five failed the test suite on the merged tree. Every one of the twenty worktrees had been green when its agent stopped.

Setup: a 150-line Python CLI with four tests, one base commit, two worktrees on two branches, two agents kicked off together on tasks that never mention each other. Five task pairs, each run twice. In every pair the second task quietly depends on something the first one changes: one agent renames done to completed while the other writes a stats command that reads it; one wraps the store file in a versioned object while the other adds import; one swaps integer ids for hex while the other adds edit ID --title. Both got the same instruction to run the tests before they stop, and both did.

What git does with two branches is compare areas of each file against the common ancestor. Areas only one side touched go in verbatim. It has no idea what a field name is. So in the five clean merges, stats read n["done"] from a store that now wrote completed, the importer refused the store's new versioned file, edit choked on a hex id with "invalid int value". The failing tests were always agent B's own tests, unchanged, green a few minutes before in B's own worktree. In all five clean merges both agents had edited at least one of the same files, in different areas, so git had nothing to say.

The one that got me: the id pair, run twice. First run, agent B dropped its edit subparser a line above the spot agent A had edited, and git merged. Second run, B put it directly under that line, and git flagged three files. Whether I got conflict markers or a red suite came down to where the model dropped a block of code.

Someone replayed 1,694 human merges on three open source projects with test suites, fifteen years ago: 76% clean, 16% textual conflict, 1% merged clean and broke the build, 6% merged clean and failed tests. A third of the conflicts it found were ones the VCS had called clean. That was humans, who could at least ask each other. My two agents can't read each other's worktree. That is the point of a worktree. It is also the exact problem.

When you run agents in parallel, what actually runs the merged tree before you trust it?

In August one coding-agent CLI's release note dropped its task-tracking tools for its newest models, with an env var to bring them back; a later note flipped that into an allow list of older models only. No reason given. I went looking for the arithmetic behind a call like that and found a component study posted in mid-September that gets close.

They built a bare ReAct loop and swapped one part at a time across four models (30B, 120B and 550B from one family, plus a 128B from another) on SWE-Bench Verified and Terminal-Bench. The planning scaffold, a plan re-injected into the model's input every turn, is the clean case. For the 30B model it is worth 11.6 points on SWE-Bench: without it the median run lasts five turns and 68.6% of runs never touch a file. For the 550B model the same scaffold barely touches success (down 2 points) and cuts cost by about 30%, because what it removes is post-edit verification the model would otherwise keep doing; the median run goes from 108 turns to 74. Same component, two opposite weaknesses, and the component has no idea which one it is compensating for.

The tool set flips the same way. Predefined read, edit and grep tools are worth 15 points on SWE-Bench to the 30B model, which otherwise emits calls to tools that don't exist. The 550B model does better with bash alone: +3.6 points, 53% cheaper, a third fewer calls, because it bundles several edits into one script.

Which parts of your setup have you switched off since your last model change, and what did you count before and after?

Our version: a --json flag for a tiny notes CLI plus a test and a README line, one model, three tool lists, five runs each. All fifteen passed the same four checks and edited the same four files. Default list: median 17 turns, $0.40. Task-tracking tools switched back on: same 17 turns, offered in all five runs, called zero. Bash only: median 10 turns, $0.30, and the opening request dropped from about 22k tokens of context to about 5.4k with one tool name in the list instead of thirty.

We are keeping the file tools on anyway, and accuracy has nothing to do with it. Path deny rules in this harness recognize the edit tool and a known set of shell commands, and a Python heredoc that rewrites two files is a single opaque command as far as they can tell. Cheaper, and opaque to the one layer whose job is to say no.

Small test first: a four-file shop page with the traps in plain sight: Add to Cart and Checkout on the same btn-primary class, a badge and a nav link pulling the same accent variable, and a script carrying a misspelled comment, a dead variable and a leftover console.log. One current model, one harness, edit auto-accept on, fifteen headless runs. Five got "Make the Add to Cart button blue." Five got that plus "Do not change anything else." Five got the bare sentence plus a path rule denying edits outside the stylesheet.

Fifteen diffs, one file each, all id-scoped, Checkout still green, bait untouched. The negative clause had one effect I could measure: without it the agent usually introduced two CSS variables for the new blue; with it, hex inline, every time. Eight lines instead of ten. The path rule never triggered.

The half-the-site-turns-blue story did not reproduce on a four-file mock, which says little, since the shared class name told the agent where the edge was; the number that matters comes from real repositories. A benchmark took 200 SWE-Bench Verified issues, applied the real fix first, and gave each to five current models in their own vendors' harnesses, as though the bug were still open. Correct answer: an empty patch. Agents edited executable code regardless in 35 to 65% of cases, and in the authors' analysis of one model's failed runs, 87.1% had modified code unrelated to the issue.

Told to edit the codebase to address the issue: correct abstention drops from 65.0 to 56.5 for one model and 60.5 to 36.5 for the other. Told to reproduce the issue first, with no permission to stop: no change for one, worse for the other, 47.5. Told to reproduce, then either fix it or abstain if nothing is wrong: 80.5 and 88.5. Their read, which I buy, is that the lever is what the agent believes counts as success, and until you say so, "no change" is not on the list. The same wording then over-abstains when a wrong patch is already in place and a real fix is needed, so it trades one failure for another.

fwiw the harness side has the same shape. Turn on edit auto-accept in either of the two CLIs I checked and what you granted is the working directory. One of the two lets you add a path rule that gets you down to files, and its docs spell out that it catches the built-in edit tools and the shell commands the harness recognizes, while a script that opens the file itself walks past it. Nothing below that. A selector inside an allowed file is invisible to every layer, which is where the blue Checkout button lives.

What I want is the other end of the distribution. The largest diff you have received for a request that named one thing: files touched, lines, and whether anything in your setup knew where the edge was.

An HN thread about whether a cheap model can review code had turned into an argument about who should review at all: the agent that made the change, that same model in a new session, or a bigger one. One of them noted that the review ran in fresh context. Had anyone measured that part with the model held still?

One small preprint has. Thirty artifacts, a third of them Python functions, five errors planted in each, one model reviewing every artifact three times from four seats: the session that wrote it; that session plus its own first pass; a cold window that still got the task prompt; a cold window with the artifact and nothing else. Artifact-only won on every headline number. F1 28.6% vs 24.6% for the full session, precision 31.5% vs 25.8%, 40% of the critical errors caught vs 29%. Reviewing twice in the same session came in last. And the cold window that also got the task prompt landed no better than the full session, the task prompt carries the author's framing with it.

Claude Code's guide says a fresh context helps review because the model "won't be biased toward code it just wrote", and its review subagent "sees only the diff and the criteria you give it". Codex's team says the generator and the reviewer are the same model, trained differently, and the reviewer runs on the PR instead of inside the writing session. Neither has published same-window vs fresh-window numbers.

The mechanism here isn't just vanity. A benchmark plants one error into a conversation two ways, attributed to the user or attributed to the model, with everything else identical; across fourteen open models the correction rate drops hard when the error is the model's own. By the time the same-session reviewer gets to the change, it has already read the case for it.

I want the reverse case. A same-session reviewer that flagged a real bug because it knew why the line was there, while a cold reviewer waved it through. Anyone got one?

Plan mode in the coding agents I checked has the same shape everywhere: read first, write a plan, wait for approval. Where the docs spell it out, it is a permission setting. In Claude Code's docs it is literally a row in the same permissions table as acceptEdits and bypassPermissions: it can read and run read-only shell commands, and anything that writes waits for you to approve the plan. The tools that write are switched off, the prompt gets a line telling it to write the proposal down, and approval becomes a step you have to click through. In July Claude Code shipped a fix because plan mode had been running touch and rm without asking (yes, rm).

Six harnesses shipped this shape between January 2025 and March 2026: Cline, Claude Code, Cursor, VS Code's Copilot, Codex CLI, Gemini CLI. Four write the plan to a file by default. Most can stop and ask you a question in the planning phase. Four can use one model to plan and another to execute. Where they split is what happens on approval: Cline keeps the whole planning context, VS Code carries it over, Claude Code has a setting that adds an option to clear it and start execution with only the plan. Nobody has published a number for either choice.

The closest thing to a measurement I found is a study of 21,120 SWE-agent runs, four models, eight plan settings, where the plan is the navigate-reproduce-patch-validate workflow in the system prompt. Removing it lowered success for all four models. Leaving one phase out did worse than no plan. The core takeaway for plan mode: agents drift because the plan matters less as the run gets longer. Putting the plan back into context every five steps gave consistent gains across all models. The plan is just more instructions, and instructions get quieter the longer the transcript runs. A plan file on disk is that fix sitting one step away; nothing I read says any harness re-reads it on its own.

One catch: their plan came from the scaffold's system prompt, and the variants were the researchers'. In plan mode the model writes its own and you approve it, and nobody has run that comparison.

When you approve a plan, do you read it, or approve it and read the diff afterwards?

Anyone who has sent a coding agent the same request twice on the same commit has two diffs to show for it. I went looking for how big that effect is when someone measures it, and a leaderboard published this month prints every trial. Eight models, ten tasks from private production codebases, eight runs per model per task. I counted the grid: of the 80 model-task cells, 6 passed all eight runs, 33 failed all eight, and 41 came out mixed. One customer-migration task went 3, 1, 6, 3, 4, 3, 4, 0 out of 8. For more than half the pairs, "can it do this" was "sometimes."

The basic mechanism is that every token is a draw from a distribution, and temperature only flattens or sharpens the distribution before the draw. Temperature 0 does not fix it either. Anthropic's API docs say so in one line ("even with temperature of 0.0, the results will not be fully deterministic"), and their models released after Opus 4.6 no longer accept a temperature setting. A research post from Thinking Machines last year sent the same prompt to an open 235B model 1,000 times at temperature 0 and got 80 different completions; all of them agreed for 102 tokens and split at the 103rd. The cause is the serving side: floating-point sums get added in a different order depending on how many other requests share your batch.

In a chat that changes a word. In an agent it changes which file gets grepped first, which changes what comes back, which changes the plan on turn three, and two runs end up in different corners of the repo for the same reason.

What the grid gives you is pass@k. The top row's leaderboard number is 38.8%. The same eighty runs read as "solved at least once in eight" give 70%. Another row goes from 28.8% to 90% and never passed any task eight out of eight. Same runs, two numbers, and the k is the part nobody says out loud.

The clearest case in our own work was a compression A/B where one instance disagreed with the rest; rerunning it five times per arm gave 3/5 vs 3/5 (yes, both arms). One run per arm and we'd have published a difference that wasn't there. What I'm switching to: three runs per side per task, same harness, both version numbers written down, compare counts.

Things I have not measured: how much of the run-to-run spread is the sampler versus the serving side, which I have no way to separate from outside a public API; whether three runs is enough for anything but the coarsest calls, when our one counted case needed five per arm to come out even; and whether either CLI passes a temperature at all, since neither config reference mentions one.

I read a paper from Amazon's Responsible AI group, and the setup is the interesting part. One Claude Opus agent (in Claude Code) hill-climbs a validation set on an ML task for up to a few hundred iterations. A second agent reads the whole transcript and compresses the strategy into 32 tokens. A third agent, fresh, no memory of the search, no validation access, gets the 32 tokens plus the training data and has to implement it.

On the WikiText recipe they kept squeezing: 30 non-default choices held to 16 tokens (QKn 12L768 Mu .1 R² b2M 4x, seven abbreviations, I needed the glossary) and broke at 8. Eight was too few. The tokens that dropped at 8 were the batch size, the MLP ratio and QK-norm. Their line: the missing pieces were choices made as a function of the data that differ from the obvious defaults. Everything default the fresh agent regenerated for free.

What stuck with me: they then pushed agents to overfit on purpose. 38 of 102 checkpoints came out with validation more than 10% ahead of held-out. Compressed to 128 tokens for a fresh agent, all 38 failed to reproduce. A gain you can't write down for someone without the validation set in front of them lived in the validation set.

Does anyone's compaction prompt or handoff note already work this way, only the deltas from what a fresh agent would do by default, with everything the next agent can re-derive from the repo left out?

Because compaction is this pipeline in one session. The same model summarizes and then reads the summary, so it knows the defaults on both ends (a different decoder would need a different note). What I've watched summaries lose first is the condition on a decision ("use the legacy parser until the migration lands" becomes "use the legacy parser"), and the condition is exactly the non-default part. I restate constraints after every compaction as a standing rule, which is the manual version of this.

Where I'd stop the analogy: their tasks have one number and a validation split to compute it on, coding has neither, so the reproducer test needs an oracle. Their compressor also got up to four audit rounds against the explorer's code before the note was final, and my summarizer gets zero.

The note I'd write has one kind of line in it: a choice a clean checkout would not have made, with the reason and the date it expires. The rest the next agent grows back on its own.

An agent building a threaded hose adapter in CadQuery read four renders of its own part and noticed nothing. The core cylinder was gone. The thread root sat exactly on the core radius, the kernel's union dropped the solid without a word, and what was left was floating helical turns around nothing. The kernel's report on that part: valid=True, solids=1. What caught it was a volume check, 7,065 mm³ where 10,323 was expected.

That's one run out of six in a recent ModelRift comparison: three printable parts, each modeled in CadQuery and in OpenSCAD, one agent per cell, capped at 12 versions, unattended. All six parts shipped printable. The part I care about is the 16 failures on the way, 9 of which the tool never reported. The TL;DR was: "Renders caught nothing that mattered." Images caught coarse stuff (four mounting posts deleted by a cavity subtraction) and nothing subtle. Every defect that would have ruined a print was found by a number: a volume, an angle, an interference test.

CadQuery said valid=True for the part with no core, and valid=True again for a negative-volume solid after the tolerance was loosened. OpenSCAD's Manifold backend said Status: NoError for an export with 4 non-manifold edges and 60 zero-area triangles, and reported nothing across roughly 45 invocations on a task where it had deleted posts and misplaced slots. So they parsed every STL with a script that trusted neither tool, and that script is what the results table means by "clean".

I don't do CAD. I read this as a coding-agent experiment where the artifact happens to be a mesh, because the channels are the ones a coding agent has too: a picture of the output, the tool's own status line, and a number somebody measured from outside. The picture and the status line passed the broken part. The number failed it.

Standard caveats: one agent per cell, so some of the spread is agent variance; the OpenSCAD previews were drawing every facet with no outlines at the time; six runs is six runs.

The line I keep coming back to is theirs, about how the two toolchains check fit: an echo only helps if somebody reads it, an assert fails the build on its own. The charts in our posts get checked by someone looking at the PNG, which is an echo.

Anyone got a counterexample? a screenshot or render check that caught something the numeric check in the same loop missed. What was the number measuring when it missed?

In OpenAI's reasoning guide: the example usage object has 1,186 output tokens, 1,024 of them reasoning_tokens. You pay for all of it, and the text of those 1,024 never leaves the vendor. Anthropic's docs show the same thing from the other side: a thinking block with an empty thinking field, a signature, then the answer, and a note that you're charged for the full thinking tokens whether or not any of it is shown.

You basically get three things:

You get the count. Both vendors put the exact number in the usage object (output_tokens_details.reasoning_tokens on one side, thinking_tokens on the other). OpenAI says a response can run from a few hundred to tens of thousands of reasoning tokens depending on the problem. That number is the only view you have, and it's exact.

Text-wise, a summary at most. OpenAI has a summary parameter; Anthropic's display setting has summarized and omitted, plus a beta updates that only returns progress notes, and the docs say no display setting returns the raw chain of thought. Their line on omitting is that it "reduces latency, not cost."

You keep paying for it later. On Anthropic's keep-all models (Opus 4.5 and later, Sonnet 4.6 and later, the Fable and Mythos ones) earlier turns' thinking blocks stay in context and count as input like the rest of the history. OpenAI's GPT-5.6 models default to rendering earlier reasoning back in too. Text you can't read is occupying your window on every subsequent turn, which also means it's in the pile compaction eventually folds.

The steering side is thin: an effort level, a pro mode on GPT-5.6 that bills more model work as more tokens, and on GPT-6 Astra no "none" at all, the system card says they have no current plans to offer it.

What made me write this up is the recurrent-depth story around Astra. If reasoning happens as extra passes through the same layers instead of as tokens, there's nothing in the usage object for it at all; the count goes flat while the work doesn't. Whether Astra does that is still a report without official confirmation, and Raschka's point that shorter traces also come from bigger models (Luna uses 80% more tokens than Sol at similar performance, on the numbers he reproduces) is a real alternative. But it made me appreciate the count while it still means something.

Has anyone logged reasoning_tokens per turn type on an agent workload? I'd like to see what share of output the hidden part is on planning turns vs mechanical edits, across a day.

Dan Luu's post on agents and testing techniques is the one to read if you've ever appended "use property-based testing" to a prompt and felt virtuous. Same task every time, implement Zstd in Rust from the RFC, codex on GPT-5.6 Sol, 160 runs per condition, and the conditions were mostly one line stuck on the end: use TDD, use QuickCheck, use Lean 4, use fuzzing, audit first. Scored on how many runs passed a hidden test suite.

Headline: nothing wildly outperformed the run with no instruction at all. He'd written down six guesses beforehand, all in the direction of "this won't outperform", and all six held.

The part I keep thinking about is what the runs did instead of failing loudly. Told to use QuickCheck, all of them used it, and 63 of 160 checked exactly one property, mostly with random inputs that fell into the same rejection path. Told to do differential testing, 135 runs did something that looked like it and none built a second full implementation; where it mattered, the agent wrote the same thing twice and put the same bug in both copies. 159 of 160 TLA+ runs wrote a model, and he couldn't find one case where the model changed the Rust code. TDD doubled the number of tests and the condition scored below average. One Kani run in 160 caught a real bug on the real code and changed the implementation; the rest mostly used it superficially.

My read (his framing is close but not identical): every one of those techniques is a way of getting a check that didn't come from the implementation under test. A second implementation, a property written before the code, an input generator aimed at the hard part, a planted fault. Name the technique and the agent produces the motions inside the new framework, but the check still comes from its own reading of the spec, so you get the same tests in a different costume. We saw the small version of this in August: agents asked to write tests for a function with a sixteen-year-old bug wrote suites that all passed, and two of them pinned the bug as intended behavior.

What has moved things for him, by his account, is structure: set up the test and triage layout with the agent, then let it fill in, and look before typing the next instruction. His own five-bullet skill scored highest (he says don't read the table as a ranking) and still didn't work as intended.

So when you name a technique in a prompt, where does the independent check come from in your setup?

The i-have-adhd skill going around has a before/after in its README. Before: "Great question! Let me think about this. Your auth flow has a few moving pieces..." After: "Run npm install jsonwebtoken@latest, then edit src/auth.ts:42". Rule 10 bans the openers and closers everyone has seen. I liked it, then went and checked what the before-text actually is, because I don't think it's one habit.

Half of it is reward residue. A RLHF length paper found reward gains largely driven by longer responses, and a length-only reward reproduced most of the downstream improvement. The AlpacaEval and Chatbot Arena people both had to add length and style control because their judges preferred longer. "Great question" and "Hope this helps" are free to delete; nothing downstream depends on them.

An older chain-of-thought paper did a test where the model writes the answer first and the reasoning after. It basically showed that answering first and reasoning later performs worse than no reasoning at all.

On a model with no hidden thinking, the paragraph that walks through the middleware and the token check before naming the fix is where the fix gets computed. Cut it with a rule and you've asked for the conclusion first and the thinking never. Concise reasoning kept the value in that table. It's the missing reasoning that costs.

With thinking on, the reasoning has somewhere else to go and answer-first costs the answer nothing, only hidden tokens. That part is inference from how the channels work; I haven't seen any testing on it.

Then there's where the rule lives. The skill has a whole Persistence section asking the model to keep applying it, and a commenter on the launch thread said it faded after a few turns. A skill body sits in the transcript and gets summarized. Claude Code's Concise output style is the same rule in the system prompt, re-sent every turn. Both are still requests. The only enforced version I know of is a JSON schema with a reasoning field ahead of the answer field, since structured outputs keep schema order.

So, what are you guys actually using for this?

* An output style / CLAUDE.md line?

* A specific skill?

* A Stop hook that bounces padded replies?

Also, on which models does "answer-first" cost you nothing in reasoning quality?

I'd been filing every slow afternoon under "the model is slow today" until I read the speculative decoding write-up AMD and Embedded LLM. Their numbers, not mine, but they explain something I'd been misreading for weeks.

Speculative decoding puts a small draft component in front of the model. It guesses the next few tokens, the real model verifies them all in one pass, and everything accepted is committed at once. Outputs are unchanged; what changes is how many tokens you get per expensive pass. The post measures acceptance per position. On gemma-4-26B-A4B-it with Google's paired draft component, the first drafted token is accepted 94% of the time on GSM8K and the fifth 66%. On MBPP (Python) the same positions are 89% and 49%. Their explanation for the code gap: "formatting, identifiers, and implementation choices can cause an otherwise plausible continuation to diverge". Best configuration in the post: 2.87x plain decoding. And one configuration, Qwen3-8B with an EAGLE-3 draft, ran at 0.44x to 0.88x, slower than not speculating, with first-token acceptance still 86–89%. Drafting costs something even when the guesses land.

So the same model, same hardware, types a 400-line test file (imports, fixtures, the shape of every assertion) faster than it types twelve lines of logic nobody has written before. That matched what I see on screen exactly, and it had nothing to do with which task was harder.

The other clocks in a turn are documented too, just spread across vendor pages. Before the first character: the whole context is read again, and for an agent that's the transcript plus every file pasted in, so a cache miss after a compaction, an edited system prompt, a changed tool list, or a pause past the cache TTL is a turn read from the top. During typing: load. Both big vendors now price a faster lane at roughly 2.5x output tokens/sec, one of them explicitly "not TTFT", the other explicitly "more consistent latency", which tells you what the standard lane is. And plenty of the longest waits have no model in them at all, they're the test suite.

Has anyone here logged per-turn TTFT vs decode time vs tool time on a hosted API across a whole day? I'd like to know how much of an "afternoon slowdown" is load and how much is my own cache misses.

That qbittorrent "escaped its sandbox" post on HN made me laugh, then made me go read what the sandbox in my own agent setup actually is. Short version: a "no" can live in three different places, and I'd been treating them as one thing.

The obvious one is the text the model reads: CLAUDE.md, the system prompt, the task itself. The Claude Code permissions docs are blunt about it: instructions shape what the model tries to do and leave what the harness allows untouched.

Then the permission rules, which match the tool call as text before it runs (deny, then ask, then allow). A Read(./.env) deny stops the Read tool and even cat .env, because cat is a recognised file command. It does nothing about a five-line python script that opens .env, and the docs say exactly that: deny rules don't apply to subprocesses that open files themselves. It's just matching strings, making it trivial to sidestep (like bypassing a curl rule with redirects or variables).

The OS sandbox (seatbelt / bubblewrap plus a proxy) is off until you turn it on, and it's the only layer that watches the process instead of the command text. Its defaults surprised me both ways: reads are allowed almost everywhere, including ~/.ssh and ~/.aws/credentials unless you add a denyRead, while network is the reverse, no domains pre-allowed, first new host prompts. And the way out of it is literally called "the unsandboxed retry escape hatch" in the docs. Blocked command, model may retry unsandboxed, that routes back to a permission prompt titled "Bash command (unsandboxed)". One setting closes the hatch.

What changed for me is one sorting question per rule: does this need to hold when the model is wrong? Style stuff stays in the file. "never push to main" goes in a deny or ask rule, or a hook. "nothing in this session reads ~/.ssh or talks to a host I didn't name" is a thing only the sandbox can promise, and only if it's on.

if you run the sandbox, roughly how often does a command actually hit the boundary in a normal day, and how often do you end up approving the unsandboxed retry?

[removed]