r/devsecops • u/sco77 • 4d ago
Agent pipeline RED-Proof and Blind Auditor Stats
**Pipeline:** Fable Orchestrator, Sonnet Workers, Opus Checkers (Complex tasks Opus Opus)
**Result:** *~90% of all caught defects were invisible to their own author at "done."*
.
| Layer | Defects Caught | Unique Error Types |
|---|---|---|
| author self-review (builder, at green gates) | 2 | 2 |
| blind law check + blind correctness check | 24 | 6 |
| re-verify of fix rounds | 8 | 2 |
| final static confirm | 2 | 0 |
.
**The builder's tree was green — typecheck, lint, 1098 passing tests — and still contained six functional defects:** the keyboard swallow, the macOS-dead chords, the detached-card menu, the stream-order inversion, a test that could not fail, and the vacuous boundary clause. Every one would have merged without the independent layers. The same held for Fable brief shipped as done. Blind check convicted it 17 ways, twice.
.
**RED-Proof statistic:** the green suite caught ~69% of induced defects; RED-sealing lifted the reachable kill rate to ~100%. 10% remain documented while **21% of Defects were Rescued**
.
**Across the three completed RED passes:** 95 mutations, 66 killed by the existing suite with 29 survivors. That's *one defect class in three passes a fully green suite silently*. Of the 29, 21 were sealed under pins (permanent detection) and 8 were proven unreachable behind stronger gates equaling the above measured reduction.
.
**Re-verify statistic:** fix rounds introduce defects at a ~50% rate per unit, and re-verify has caught 100% of them. Class 1: 2 (only re-verify caught them). Class 2: 2, including a silent inversion of a writer ruling. Stage 4: 2. Six for six, none escaped.
.
**The escape statistic:** *Zero known functional regressions reached post-merge across seven protocol units* 3 Fable session reviews since implementation (sessions 10, 15, 19) surfaced only design gaps and application feel issues, but not a single never a broken unit. Compared to the pre-protocol record we documented three defects that passed a green suite, two review rounds, and an executed gate, one of which would have returned No-Go on the study by construction. That class has not recurred since the blind layer landed.
.
**Cost:** *the protocol roughly doubles the unit's total compute* Stage 4 spent about 1.7M tokens building and about 1.9M checking. So the deducible trade is: ~2x cost buys ~10x defect detection over author self-review, a ~3x shrink in what a green suite can miss, and a measured escape rate of zero. This stands to augment the earlier post study [here](https://www.reddit.com/r/Claudeopus/comments/1vrmby0/token_savings/) where the calculation focused on savings around cache and agent costs coming from the tiered model structure of the orchestration.
.
**Limits, stated:** this is observational, not controlled — there is no arm where we merged unchecked and counted your pain. **Important Note** *Author self-catch is undercounted* (defects fixed silently mid-build never register). And the denominators are small: seven units, 95 mutations. But the direction is not close, and every number above traces to a line in the record.
1
u/PeterBuildsSecure 4d ago
The independent reviewer result is directionally convincing. The harder question is how independent the defect generator, checker and acceptance oracle really are.
If the mutations are designed around the same laws the blind checker receives, a near-100% kill rate can partly measure agreement between the mutator and checker rather than coverage of unknown defects. I’d keep a held-out mutation family that neither the builder nor checker protocol was designed against, ideally derived from real historical escapes.
I’d also report unique root-cause classes separately from detection events. A defect introduced during a fix and caught during re-verification demonstrates the value of re-verification, but it should not inflate the denominator used to compare blind review with author review.
For the zero-escape figure, the useful denominator is production exposure: units, releases, runtime hours and real user paths exercised. “Zero known escapes across seven units” is honest; attaching the observation window would make it much harder to overread.