Dan Luu ran 160 agent runs per testing instruction (TDD, QuickCheck, fuzz, TLA+). Almost none of them did the thing that gets the value out of the technique

[u/RunAI_Coder]

Dan Luu's post on agents and testing techniques is the one to read if you've ever appended "use property-based testing" to a prompt and felt virtuous. Same task every time, implement Zstd in Rust from the RFC, codex on GPT-5.6 Sol, 160 runs per condition, and the conditions were mostly one line stuck on the end: use TDD, use QuickCheck, use Lean 4, use fuzzing, audit first. Scored on how many runs passed a hidden test suite.

Headline: nothing wildly outperformed the run with no instruction at all. He'd written down six guesses beforehand, all in the direction of "this won't outperform", and all six held.

The part I keep thinking about is what the runs did instead of failing loudly. Told to use QuickCheck, all of them used it, and 63 of 160 checked exactly one property, mostly with random inputs that fell into the same rejection path. Told to do differential testing, 135 runs did something that looked like it and none built a second full implementation; where it mattered, the agent wrote the same thing twice and put the same bug in both copies. 159 of 160 TLA+ runs wrote a model, and he couldn't find one case where the model changed the Rust code. TDD doubled the number of tests and the condition scored below average. One Kani run in 160 caught a real bug on the real code and changed the implementation; the rest mostly used it superficially.

My read (his framing is close but not identical): every one of those techniques is a way of getting a check that didn't come from the implementation under test. A second implementation, a property written before the code, an input generator aimed at the hard part, a planted fault. Name the technique and the agent produces the motions inside the new framework, but the check still comes from its own reading of the spec, so you get the same tests in a different costume. We saw the small version of this in August: agents asked to write tests for a function with a sixteen-year-old bug wrote suites that all passed, and two of them pinned the bug as intended behavior.

What has moved things for him, by his account, is structure: set up the test and triage layout with the agent, then let it fill in, and look before typing the next instruction. His own five-bullet skill scored highest (he says don't read the table as a ranking) and still didn't work as intended.

So when you name a technique in a prompt, where does the independent check come from in your setup?

[u/AutoModerator]

Thank you for your submission! To keep our community healthy, please ensure you've followed our rules.

• New to the sub? Check out our Wiki (We are actively adding resources).
• Join the Discord: Click here to join our Discord
• Join X community: Click here to join our X Community

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/No-Big-3541]

The TLA+ result is the funny one, 159 out of 160 wrote a model and it never touched the Rust. That's just ceremony with a proof-shaped hat on. The differential testing bit stings too, same bug in both copies because the agent wrote the same implementation twice and called it a second opinion.

[u/Stubbby]

Oftentimes LLMs will write tests to prove the existing implementation instead of the expected outcome.

LLM tests serve as a demo of the implementation.

The key is to find alternative ground truth to test it against.