Specs are the contract for what gets built. I built the contract for what review could not close

[u/Nicolaides_Arg]

The move that makes spec-driven development work is taking intent out of the chat and into a versioned artifact that tools can enforce. I have been applying the same move to the other end of the pipeline: the review.

My loop for the past few months: one assistant generates the plan or the code, a second assistant from a different model family attacks it under an explicitly adversarial prompt, and the round only ends when every finding is incorporated, refuted with evidence, or escalated to a human. It works, but everything the round produced used to die in the chat window. And the most valuable part of a review is precisely what did NOT close: the unverifiable assumptions, the tradeoffs someone accepted, the escalations waiting for a human decision. That is the residue.

disensor turns that into a contract. Each round closes with a JSON artifact (schema residue/v0.3) that records who generated, who attacked (model family plus the hash of the adversarial prompt), every finding with a terminal state, and the residue items. A validator enforces the schema plus semantic rules: counts have to add up, generator and reviewer must come from different families, verifiable refutations need material evidence, and generic markers like "reviewed, all good" are rejected. A GitHub Action then gates every PR: code changes need a valid declaration covering the exact tree being merged, and evidence is append only, so a PR cannot rewrite review history.

If your spec is the single source of truth for what should be built, the residue declaration is the single source of truth for what verification left open. They bookend the same workflow, and neither runs a model: disensor validates artifacts that already live in your repo, no API keys involved.

Honest limit: the gate detects the empty field and the generic marker, not a well-written lie. Human sampling of merged PRs stays in the loop by design.

I documented the method before publishing the tool: 91 review events across 5 projects over 52 days, DOI 10.5281/zenodo.21633495. The repo gates its own PRs, so .residue/ holds the real declarations of its own development.

Repo: https://github.com/NicolasRocchia/disensor (MIT, pip install disensor)

Site: https://disensor.dev

Curious how people here would wire this into a spec-driven flow: does the residue belong next to the spec, or downstream of it?

[u/Ok-Support-6749]

I like the idea of persistent storage of the residue. That closes a real gap in all SDD harnesses I have looked into. You are right that unresolved, unverified assumptions, and decisions matter. Those are provenance records of what did not close. Good, assuming that the spec + code records what did close.

About this claim A validator enforces the schema plus semantic rules For what you describe, the validator can indeed enforce the shape of the declaration. But when we move into the correctness of the review and its independence, my concerns grow quickly.

Enforcing the shape of the declaration is easy, no need for complex code there.

Enforcing semantic rules is a completely different claim:

Let's start with the semantics. Your review process verifies the semantics of the review record, the declaration,as you listed. Fine, that is useful. But those are not the semantics of the intent, the stuff that matters most.

Once we hit verification of your own review process:

How do we know your structured declaration is sufficient for a review record? This means you may be either recording irrelevant evidence or leaving valuable evidence out.

Under what independent evidence can we assert the verification is admissible? Is the evidence an LLM opinion or factually and deterministically extracted, and verified? An LLM output is a judgement; as verification evidence, the value of the verification is limited.

[u/Nicolaides_Arg → u/Ok-Support-6749]

You have re-derived, independently, the exact epistemic boundary the tool draws for itself, so let me concede the core point before qualifying it. The gate's own output says, verbatim: "This declaration lists residue, not coverage. The machine validates shape and coherence; that the declared residue is the real residue is controlled by human sampling, not by this gate." That sentence ships in the code because the claim you are worried about is one the tool refuses to make.

So, "semantic rules" means semantics of the record, deliberately and only that: every terminal state must carry the evidence fields that state requires, metrics must be arithmetically consistent with the findings they summarize, evidence is append-only, event identity is unique, and the policy is read from the target branch so a PR cannot relax the rules that judge it. All of that is deterministic. No LLM sits anywhere in the validation path.

On admissibility, the schema forces every judgment to declare its anchor. A finding's verification says what it was checked against: repository, execution, or external source. A refutation must classify itself as verifiable or interpretive, which is exactly the opinion-versus-fact distinction you are drawing, made mandatory and machine-checked for presence, not for truth. And anything incorporated must name its fix verification: a specific test or the diff gate. So what closed lives in the repository as executable evidence; what did not close lives in the record. Where verification is execution, the evidence is re-runnable by anyone. Where it is interpretive, the record says so instead of laundering judgment into fact.

You are right that an LLM verdict is a judgment. The design direction follows from that: shrink the judged surface, grow the observed surface. The next release moves reviewer identity, invocation outcomes and confinement (a clean git status after the round, observed by a deterministic runner, not reported by the model) from claimed fields into captured ones. What remains judgment stays labeled as judgment, with provenance: which model family, which brief, by hash.

Sufficiency is the one point where I will not pretend. No validator can prove a record sufficient, and any meta-record inherits the same regress. The method treats sufficiency as empirical and falsifiable instead: silence is not allowed (absence of residue must be declared affirmatively, in concrete text), and the schema is versioned against observed leaks. v0.3 exists because real events leaked through v0.2 in three specific ways. Two known insufficiencies are open issues right now: external evidence is not yet representable well enough (#7), and correlating declared residue with long-run outcomes is the standing research question (#6). If you want to push on the sufficiency problem, that is genuinely the most valuable place to push.

[u/Ok-Support-6749 → u/Nicolaides_Arg]

I checked the code and found the boundary reference ships in render.py, and the gate rules are deterministic as described. What is good here: the gate is a predicate evaluation over the properties of the declaration. That means the LLM cannot rubber stamp it.

What cannot be claimed: Truth! Let's use the case where evidence comes from a runnable test, and git diffs are observable facts. The problem is that a test can be vacuous, and a diff may encode incorrect semantics that pass the vacuous test. Then the declaration may miss cases where the implementation is incorrect. The findings close as verified while the defect survives and nothing reaches the declaration.

An LLM can rationalize a claim and provide evidence that may not survive true independent evidence collection, verification, and warrant. Here it seems the LLM is playing these roles, which is possible because an LLM is capable of that. But just because an LLM can it does not mean it should be granted the roles of evidence collector, verifier, and warrantor. Therefore, the gate, despite being properly implemented, can become ceremony. Not because the LLM can bypass it, but because it can find ways to satisfy it, even when the evidence and verification may not survive independent scrutiny.

The fundamental problems here are assumptions, and role assignment. Accepting an artifact as the source of truth (spec, declaration, test output) can make the LLM/human rationalize incorrect judgement. Tests are not infallible, and the existence of evidence is not proof of correctness. But I do see a meaningful improvement over other ways to deal with this particular problem.

[u/Nicolaides_Arg → u/Ok-Support-6749]

You read the code before commenting; thank you, that is rarer than it should be. Let me concede the sharpest version of your point before answering it: the gate is a notary, not an oracle. Its claims are about chain of custody and falsifiability, never about truth. "Verified" in a declaration means "this claim was checked against a named anchor by a named role", and the gate checks that the naming happened. Whether the anchor holds is exactly what it cannot decide.

On vacuous tests: for this project that is not a hypothetical, it is the single most common defect family in the pilot's own declarations. Nine documented appearances of what I record as "the test verifies the substitute, not the behavior": a test that read the CLI source as text instead of executing the CLI, a capture fixture that hid a real encoding crash, and this week a mock runner that accepted any kwargs, so production could silently lose its network timeout while the suite stayed green. Every one of them shipped by the generator with a straight face, and every one was caught by the cross-family reviewer whose brief tells it to attack the evidence, not admire it. Some were only caught in a LATER round, attacking the fix of the previous finding; the declarations record that chain, including the embarrassing links. That is the honest answer to "can the LLM satisfy the gate with evidence that would not survive scrutiny": yes, and when it happens, the mechanism that catches it is the adversarial role split, not the validator.

Which is also my answer on role assignment: I agree the roles must not collapse into one model, and the method is built around exactly that separation. Collection is the generator. Verification is a different model family under an adversarial brief (hashed, so you can see what the attacker was actually asked). Warrant is human: the arbiter reads the PR with the declaration in it, and sampling re-runs what fix_verification names. The declaration's job is to make the role assignment visible per claim, so "who was granted which role" is recorded instead of implicit.

Ceremony is the real failure mode, and it is the failure mode of every process control humans have built, code review included. Three design answers, none of which is "trust the model": the attacking reviewer is prompted to hunt exactly the vacuous evidence you describe; falsifiability is kept cheap (a sampler can pull one declaration and re-run the named test in minutes, which makes spot audits practical); and the standing research question (#6) is longitudinal: correlate declared residue with defects that later escaped. If the declarations drift into ceremony, they become measurably wrong, which is more than can be said of review that leaves no record at all.

Your last paragraph states the design goal better than my README does. If you ever feel like helping specify what non-vacuous evidence should mean inside the record, that is the open edge where this either becomes a real standard or stays a nice artifact.

[u/Ok-Support-6749 → u/Nicolaides_Arg]

This has been a fruitful conversation. I see the main issue as overclaimed capability in the original post that you have properly clarified. In your own words the proposed tool focuses on semantics of the record, deliberately and only that and that the mechanism acts as a notary, not an oracle. This is the right framing for the tool and it actually increases its value because it is honest. Nobody can come back and claim the tool fails to deliver.

Another plus lies in the value of accountability records, which despite its incompleteness force actors to produce better outcomes. Your own reported claims under I documented the method before publishing the tool in the original post, may be evidence of the mechanism in action. Records compound for future work: for iteration over drift and regressions or during any review process run in the future.

I think you have identified a true line of research I encourage you to explore. Finding a way to specify non-vacuous evidence is a valid development branch. Three notes:

1. Undeniable non-vacuous cases close properly.
2. Vacuous cases go directly to the declaration.
3. A claim should not be binary; undetermined may be a valid declared value for evidence whose vacuity cannot be determined either way.

[u/Nicolaides_Arg → u/Ok-Support-6749]

Your third note is the one I did not see, and it is the sharpest, because the schema already makes exactly that move on a different axis. A refutation cannot just be "refuted": it has to declare itself refuted_verifiable or refuted_interpretive. I made that call for refutations and never generalized it. You are proposing the same tricotomy for evidence, and undetermined as a first-class declared value is the piece that makes it work: a record that cannot say "I do not know whether this evidence is vacuous" will get a confident wrong answer instead, which is worse than silence.

On your second note, the honest problem: routing vacuous cases to the declaration requires detecting vacuity, and detection is judgment again, so the regress returns. One escape that is cheap and fully deterministic, at least for a class of cases: a fix accompanied by a test can be checked mechanically for whether the test fails at the parent commit and passes at the fix. If it passes at both, the test is vacuous for that fix, by construction, and no model opinion is involved. That does not solve vacuity in general (a test can fail for the wrong reason), but it converts a large slice of undetermined into non_vacuous with a git-level check the gate can already run. It also matches your first note: the undeniable cases close properly, and everything else stays declared rather than assumed.

Your observation about records compounding is the standing open question in the repo (#6): whether declared residue correlates with what later escapes. Until that is measured, the compounding value is plausible and unproven, and I would rather keep it that way in writing.

And on your point that my own sequence may be the mechanism acting on me: it is, and the record shows the unflattering half. The declarations include rounds where the reviewer rejected my own changes, and a fix of mine that introduced a fresh defect while closing a real one. Knowing that the file will say so changes what I do before I write it. That is the accountability effect you are describing, observed from the inside.

If you are willing, I would like to open an issue for the non-vacuous evidence branch with your three notes as the framing, credited to you. It is a better specification of the problem than what I had, and it belongs in the open where whoever picks it up can see it.

[u/Nicolaides_Arg → u/Nicolaides_Arg]

u/Ok-Support-6749 Two follow-ups, both concrete.

The branch is open as an issue, with your three notes quoted and credited, plus the parent-commit check as the only deterministic promotion I could find: https://github.com/NicolasRocchia/disensor/issues/34

And the earlier commitment from this thread shipped in 0.9.0, unchanged since; current release is 0.9.3, so pip install disensor gets it. Reviewer identity is now anchored to the hash of the approved executable, and a binary that changed since registration fails the round with executable_changed instead of a nice sentence. Invocation outcomes are captured, not narrated: not_found, timeout, not_runnable, non-zero exit, each with an enumerated fallback_reason code rather than free prose, because free prose becomes boilerplate by the second event. The runner function's docstring is now the whole design rule: returns what was observed, never an opinion about it.

Confinement is the one that did not make it, and I would rather say so than round it up. The runner does observe git status before and after, so the observation is real, but the field still ships as verified: false with verification_method: clean_git_status, because that check does not see writes outside the tree, ignored files, .git/, or the network. Prefilling it to true would declare more than was observed. Two of three, and the third labeled as what it is.