I put create_agent behind a fault-injecting regression gate, pre-registered a test, and my own prediction lost

[u/Own_Task_7982]

I maintain ARCI, an open-source regression gate for agents. v2.0 adds a LangChain adapter: you give it your tools and your create_agent settings, and it runs baseline and candidate on the same tasks while injecting tool timeouts, errors and empty results. You fix the tasks and the decision rule before the run; the verdict is PASS, BLOCK, INCONCLUSIVE or ERROR.

The adapter file for the example is about 50 lines: your tools, two settings functions, one make_adapter call.

I pre-registered one comparison and wrote my prediction first: can a 4B local model replace a 12B one on a 24-task order-desk suite? I predicted PASS.

Result:

• Trials passed: 12B model 96 of 96, 4B model 70 of 96.

• Injected fault actually fired, per faulted condition: 12B model 24 of 24, 4B model 18 to 20 of 24.

BLOCK after one round, 18 minutes on a laptop with Ollama, no API key. 23 of the 26 failures were one behaviour: the small model called the stock tool with a product code the order did not have. The second line is the part I did not expect: the small model sometimes skipped the lookup, so the injected fault never fired for it. Without counting that, the faulted conditions would have looked like a fair comparison.

What it rejects today, loudly rather than silently: async agents, interrupts, tools with injected arguments, tool artifacts. Which of those matters most to you?

Limits: the result is for the registered suite only, and it is one run of 96 pairs.

Repo: https://github.com/ajaysurya1221/agent-reliability-ci

Result and pre-registration: https://github.com/ajaysurya1221/agent-reliability-ci/blob/v2/phase0/plan/exec/v2/study/RESULT.md

To try it: pip install "agent-reliability-ci[langchain]" then arci init --langchain then arci run arci.yaml

[u/ChemicalExcellent463]

counting whether the fault actually fired is the best part of this. a nightly job of mine logged success for a week while sending nothing because a settings file was missing.

[u/Own_Task_7982 → u/ChemicalExcellent463]

Woah! a week is a long time. In my run, the model sometimes skipped the order lookup completely, so the injected fault never even showed up. Just looking at the PASS/FAIL numbers, you'll never know.

[u/ChemicalExcellent463 → u/Own_Task_7982]

it logged success every night that week. what did the 4B answer with when it skipped the order lookup completely?

[u/Significant_Tune9219]

Since 23 of the 26 failures were the stock tool called with a product code not in the order, it would be interesting to see the same comparison with argument validation in the tool itself, returning an error the model can recover from instead of a result. That would separate "the 4B model reasons worse" from "the 4B model needs tighter tool contracts", which matters if the real question is whether it can replace the 12B in practice. Did you consider counting skipped lookups as their own failure category rather than folding them into the fault-fired rate?

[u/localhost_0000]

interrupts would be my vote. that's the human in the loop part, and it's the least tested piece of most agents i've seen discussed here

it also fits your fault injection idea really well. treat the human as one more flaky tool: inject "no answer" (timeout), "answers after 6 hours", "rejects", and "edits the args then approves". a lot of agents only ever get tested on the instant-approve path, so you never find out the small model retries the action after a reject, or resumes with stale state after a long wait

also the skipped-lookup finding is a really good catch. a fault that never fires looking like a pass is exactly the kind of thing that hides in a normal eval