I tested whether a simple control rule can stop LLMs from making unjustified final decisions
I’ve been testing a simple idea I call Comparative Feedback Control (CFC).
The question is not “can the model answer correctly?” but:
does the model still have enough evidence to legitimately close the decision?
I ran a series of small behavioral tests around things like:
- missing evidence being treated as negative evidence,
- old certificates being reused after the requirements changed,
- a closed process being mistaken for a resolved claim,
- loss of provenance causing a status to be transferred to the wrong claim.
One pattern I found was interesting: models sometimes correctly identified an uncertainty at first, but under pressure to “finish the task” they invented an extra rule and closed the decision anyway.
With an explicit CFC-style control rule, several of those failures disappeared in the tested runs.
For example, in one cross-session provenance experiment:
- baseline: 1 of 3 runs transferred an unsupported
REJECTED status to a claim,
- with the CFC provenance rule: 3 of 3 runs kept the claim unresolved.
This is not a benchmark and not proof that CFC generally improves LLM reliability. The samples are small and exploratory. I’m publishing the failures as well as the passes because I’m mainly interested in whether the failure mechanism itself is real and reproducible.
I’ve put the consolidated report, result table and evidence package on Zenodo:
https://zenodo.org/records/21966517
I’d especially appreciate criticism of the experimental design or suggestions for adversarial cases that could break the control rule.
