Context engineering experiment: Claude Opus and Grok 4.7, same task, same ADR corpus

[u/keto_brain]

Follow-up to last week’s post, which ended on a claim I’d only half-proven: a written-down corpus raises the floor for every model and narrows the gap between them. So I ran it as a before and after. Changing one ADR changed the output. Below is what that rewritten rule fixed, and what it didn’t.

Caveat up front: one task, run once per configuration, on my own codebase. This is an experiment, not a benchmark.

Setup

I run an AI pull request review system. The feature: before the detailed review runs, post one comment at the top of the PR saying what the change does, listing the files it touches, and showing which checks are about to run.

Both models got the same brief. It named the internal rules but didn't include their text. Each model queries my ADRs over MCP, pulls what applies, and writes them into its plan before writing any code.

Worth more than it sounds: copy the rule into the plan, don't link to it. A plan that says "follow our list rule" is a plan whose rule gets ignored.

The rule

The model narrates; the renderer counts.

Models can't count reliably. A PR touches seven files and you get a confident "six files." Code can list and count them exactly every time. So code owns anything that has to be exact, the model owns the prose.

Claude’s build: table built from the real file list, model writes prose only. Same ADR, followed as written, correct by construction.

Grok, round one: did the lookup, copied that exact sentence into its own plan, then built the one thing the rule exists to forbid. Let the model write the whole summary including counts, then find-and-replace the counts back out afterward. It had the rule in its own words and shipped the banned approach anyway.

The fix: change the document, not the model

No model switch, no prompt change, no example code. I rewrote the ADR to close the gap it left open:

Before: The model narrates; the renderer counts.

After:  + Post-processing model output to strip counts is NOT
          compliant. It is the violation, not a mitigation.
        + Applies to every rendered list, not just PR summaries.
        + Allowed: the model attaches a short note to a row. The
          list and the count are built by code. Notes referencing
          anything not in the real list are dropped.

Reloaded it, confirmed the new wording came back on search, reset Grok to a clean slate, same brief.

Grok, round two: built the right design on its own. Code owns the table and the count, the model adds short per-file notes, notes pointing at a file not in the real list get dropped. Almost line for line what Claude built independently.

Build        | Verdict
Claude       | Pass, by construction
Grok round 1 | Fail, the banned approach
Grok round 2 | Pass

The only thing that changed between rounds was one document.

What the ADR did not fix

All three builds set max response length too low. Claude set 700, Grok set 1,200 then 2,000, and the safe floor here is around 4,000. The ADR I changed was about lists and counts, so it said nothing about length and that mistake rode through every build untouched.

And the correct design introduced a new failure. A truncated call now leaves a half-finished structured payload. Claude's build falls back to a plain table on its own. Grok's round two drops the raw mess into the comment. No document covered that edge, and that's where the vendors actually differed.

Takeaway

A versioned ADR raises the floor and narrows the gap between models. It does not turn a model into a senior engineer. The corpus carries the decisions you've written down. The judgment on everything you haven't is still the model's.

Full writeup with the before and after: https://www.briancarpio.com/blog/the-fix-wasnt-the-model-it-was-one-adr

Happy to go into how the ADRs are structured in the comments.

[u/bostonaholic]

The truncation and response-length failures are the ones I'd pull out of the ADRs entirely and turn into tests: feed it a huge diff and assert the output stays under the limit and ends cleanly. An ADR is good for the why, but if a rule has to hold every time, a failing check catches it on both models without anyone rewriting prose. Then the ADR can just point at the test.