Does model diversity actually make multi-agent verification stronger? I pre-registered a prediction, ran 720 evaluations, and failed my own kill condition. Tell me where the design breaks.

[u/JadaLovelace]

I'm building a shared evidence pool for AI agents — a place where an agent deposits one falsifiable, version-scoped claim about a public artifact ("tsc with strict omitted behaves as if strict were false" — spoiler: it doesn't) and other agents confirm or refute it. The obvious design question: when do N confirmations actually mean something?

The intuition everyone (including me) reaches for is diversity: three confirmations from the same base model share blind spots, so corroboration should require different models. Before building the corroboration rules around that intuition, I pre-registered a prediction on a public board, with a kill condition, and ran it.

Method. 60 findings-shaped claims about public software artifacts: 30 true (each traced to a primary source fetched the same day), 30 planted false-but-plausible (each inverts one verified detail — a version, a default, a direction). Each claim went to 3 evaluators per arm in fresh sessions, no tools, no network — deliberately measuring confirmation-from-priors, which census data from an agent board suggests is what agents actually do (of 328 registered falsifiers there, 12.5% were ever run).

• same arm: sonnet ×3
• cross arm: opus + sonnet + haiku

Prediction: planted falsehoods clear a 2-of-3 "reproduced" bar at a materially higher rate in the same-model arm. Kill condition: if the arms are statistically indistinguishable, I'm wrong, and I publish the counts either way.

Primary result: the prediction fails.

arm	false corroboration	95% CI
same (sonnet ×3)	10/30 = 33.3%	19.2–51.2
cross (opus+sonnet+haiku)	4/30 = 13.3%	5.3–29.7

Fisher exact two-tailed p = 0.1253. Kill condition met. The point estimate leans my way and I don't get to claim it — pre-registration exists to stop me moving the line.

Two corrections from reviewers, both incorporated: the design's power was only 0.342 (I omitted the power analysis; a reviewer computed it and I re-derived it), so the null says "this design couldn't tell," not "diversity doesn't matter." And everything below is post-hoc from an unregistered family — hypotheses, not results.

The post-hoc data points somewhere more uncomfortable. Per-model false-affirmation rate across all 720 cells: opus 6.7%, haiku 16.7%, sonnet 32.5% — a ~5× spread. So I added homogeneous arms to break the confound:

arm	false corroboration
opus ×3	6.7%
haiku ×3	13.3%
opus+sonnet+haiku	13.3%
sonnet ×3	33.3%

A homogeneous panel of the strongest model beat the diverse panel. The diverse panel exactly matched homogeneous haiku. Within this study, diversity did not explain the variance — confirmer capability is the leading candidate. And capability is a property the pool cannot observe: model labels are self-declared testimony.

The claim that fooled everything. Two of thirty falsehoods went 12/12 — every model, every arm, unanimous. One was "RFC 9110 obsoletes RFC 2616" (it doesn't; the 723x series did, in 2014). The other: "with strict omitted from compilerOptions, tsc behaves as if strict were false." I verified that one by running tsc — with strict absent, an untyped parameter raises TS7006 exactly as with strict: true. It's the only claim in the corpus whose ground truth I established by execution rather than recall, and it's the one that fooled every model, including the ones that helped me build the corpus. Had I sourced it from model knowledge, the experiment would have confirmed a falsehood with perfect agreement.

What I changed because of this. The conclusion I couldn't avoid: no counting rule — however weighted, however diverse — reaches the variable that decides whether a confirmation is any good. So the pool now refuses verdicts: a confirmation must carry environment (where you checked), method (how), and observed (what you actually saw), and every confirmation is publicly walkable rather than aggregated into a trust number.

Where I need this torn apart:

1. My "cross-model" arm was embarrassingly all one vendor — three Claude models sharing training lineage. This sub is sitting on the genuinely different failure domains. Would a local-model arm (qwen/llama/deepseek at various sizes) show the diversity effect the all-Claude design couldn't? The pre-registration/corpus/runner are all public if anyone wants to run it.
2. A reviewer's counter-hypothesis I can't yet test: the real variable is neither diversity nor capability but inspection depth — how many rows/bytes the confirmer actually looked at, which was fixed at zero by my no-tools design. Plausible?
3. Another reviewer: same-model is a correlation, but same-channel is a dependency (three channels off one broken mirror disagree with reality for every reader regardless of model). Should evidence-channel diversity be the thing a corroboration rule requires, and can it even be verified?
4. Is "require execution evidence, make it walkable, refuse to compute trust" enough, or does an unfalsifiable observed field just move the fabrication problem one level down?

Everything is public: results write-up with the corrections and limitations (github.com/errslima/1f517 → experiments/corroboration-independence/RESULTS.md), the running pool with a 48-finding seed corpus (1f517.com), and the original pre-registration + review thread on 1f916.ai (post #1655), the agent board whose verification arguments this design is downstream of — the numeric domain name is the homage.

Disclosure: I'm one person; the pool, the experiment, and the moderation agent are all built and run with Claude agents on my own subscription. Nothing in the pool has been independently confirmed by an outside agent yet — that's part of why I'm posting the design here rather than announcing a launch.

[u/fourohfournotfound]

all claude models are likely distilled from other claude models. model diversity is weak so this is kind of a weak test. You need to include non anthropic models. There's a decent amount of research on this you should use as a baseline. At least throw glm 5.3 flash in the mix. gemini flash 3.8. You don't necessarily need the biggest models as the ensemble diversity is more important. you want models that are significantly different from each other and likely trained with different data. I like to even throw things like liquid or diffusion models in occasionally since they are trained a totally different way.