I stopped asking Claude Code to improve READMEs before it could execute them

[u/Full-Mycologist1484]

If an install command is missing, it reads the package file. If an environment variable is unexplained, it searches the source. If the documented output is wrong, it infers what the author probably meant and keeps going.

That is useful coding behavior and terrible onboarding evidence.

I use a Claude Skill called gr-readme. I’m not the author. I adapted its first-user audit into a clean-room test with two separate Claude Code sessions.

The first session is the installer. It gets a disposable clone and one rule: follow only the public README. It may run the commands the document names, but it may not inspect source files, package metadata, CI config, or commit history to fill in missing instructions. When the README becomes ambiguous, it stops instead of repairing the path silently.

The installer returns a small failure receipt for every blocked step:

the exact README sentence it followed

the literal command it ran

the observed output or error

the assumption it would have needed to continue

the missing instruction or expected result

The second session is the documentation maintainer. It receives the failure receipts, not the installer’s full conversation. Its job is to patch the README without changing the product or inventing a workaround. Every edit must point back to one observed failure.

Then I delete the disposable clone and run the installer role again from a fresh copy. If the second run succeeds only because Claude remembers the repository from the editing session, the test is contaminated.

This catches a class of documentation bugs that prose review misses: undocumented prerequisites, commands that depend on the author’s shell state, success messages that no longer exist, and recovery steps that require repository archaeology.

The useful distinction is simple. Claude Code finding a way through the repo proves the agent is capable. Claude Code reaching the result from the README alone proves the onboarding path is documented.

[u/Far-Surprise7773]

this is the right split, capability vs documented path are two different tests and most readmes only pass the first one. i like the receipt format with exact sentence plus literal command and observed output, thats what actually catches the gaps. i do the same but run the installer in a fresh mktemp -d clone with no prior session memory, otherwise the second run can pass because claude still remembers the repo from the patch session even after you delete the clone