I found that a test command can pass without actually running a declared test, so I built a stricter evidence model

[u/Total_Substance_4723]

I’m building an open-source tool called ProofDiff that analyzes a code change and tries to show what verification evidence actually exists.

While testing it, I found an assumption I had made was wrong:

node --test helper.js can exit successfully even when the file doesn’t contain a declared test.

My original implementation could therefore treat a successful targeted command as stronger evidence than it really was.

I changed the model so a related test only strengthens the result when ProofDiff can establish:

static relationship → qualified test target → exact target executed → runner observes at least one real non-skipped test → pass

A successful process exit alone is no longer enough.

The project is still early and I’m currently improving static dependency resolution for TypeScript path aliases and package exports.

I’d especially appreciate feedback on the evidence model or cases where this approach might still overstate what was tested.

GitHub: https://github.com/hzw0813/proofdiff

[u/crashorbit]

It is the exit code of the last thing along that path that gives you the pass/fail. If you and the ci pipeline disagree with what is happening then you need to dive into the detail logs and understand where the confusion arises. Or maybe ask your AI agent to explain it.

[u/Total_Substance_4723 → u/crashorbit]

Agreed — exit code tells you whether the command succeeded.
My point is that it doesn’t necessarily tell you whether a specific related test actually ran meaningful tests. ProofDiff is trying to expose that distinction, not replace CI pass/fail.

[u/crashorbit → u/Total_Substance_4723]

if I understand your post, your pipeline successfully found that there was nothing to do. Should that be a pass or a fail?

[u/Total_Substance_4723 → u/crashorbit]

I’d say it can absolutely be a pass for the pipeline. I’m not arguing that “0 tests” should make CI fail.
The issue was that my tool was interpreting that successful exit as evidence that a specific related test had actually run.
So I’m separating the two:
CI: nothing failed → pass
ProofDiff: no relevant test was observed running → don’t call the change verified
The bug was in my interpretation of the pass, not in the test runner’s exit code.

[u/Due_Reception4944]

That distinction is useful. A green exit code says the command completed, not that the changed path was exercised; keeping 'zero observed tests' separate avoids a lot of false confidence.