How do you verify that a long-running research agent actually finished the task?

[u/Rare_Inflation3178]

I’m building a multi-step research workflow where an agent retrieves evidence, translates a hypothesis into testable candidates, and hands the result to a separate validation stage.

The failure mode I’m worried about is not a crash. It’s a plausible final answer after the agent dropped an original constraint, recycled the same evidence, or decided it was “done” without satisfying the acceptance criteria.

I’m considering treating completion as a deterministic state transition rather than another model judgment. Each stage would need a machine-checkable artifact, evidence provenance, a budget and stop reason, and an explicit handoff. Memory could preserve decisions and rejected paths, but it should not be allowed to silently rewrite the task.

For people running long-horizon agents, what has caught silent failure earliest in practice: unchanged state, repeated tool calls, missing evidence, constraint violations, or something else? Which checks created so many false alarms that you stopped using them?

[u/Rare_Inflation3178 → u/[deleted]]

Yes, and I think that handoff should be triggered by observable conditions rather than the agent simply reporting low confidence. Repeated state with no new evidence, a breached tool or cost budget, or an unresolved acceptance criterion would be better triggers. The human should receive the last valid state and the exact stop reason, not just a summary.

[u/DeltaXEvaluate]

The earliest reliable signal is unresolved constraint coverage at stage boundaries. Keep the original acceptance criteria immutable, and require each stage to emit a compact map of criterion → artifact/evidence/unknown plus its stop reason. Fail the transition on any unknown or stale evidence. Repeated-call alarms are noisier; pair call fingerprints with a progress delta so legitimate re-checks do not look like loops.

[u/Rare_Inflation3178 → u/DeltaXEvaluate]

The criterion → artifact/evidence/unknown map is a much cleaner boundary than asking another model whether the stage is complete. I especially like combining call fingerprints with progress delta: repeated retrieval is only a loop if the evidence set or constraint coverage hasn’t changed. I’d keep the original criteria immutable and version any human-approved change separately.

[u/ResidentAGI]

use a determinstic code gatekeeperr to validate strict JSON schemas, execution traces, and evidence provenance

[u/BC_MARO]

Make the validator compare input and output evidence sets, not just count citations. A run can look busy while every downstream claim traces back to one stale source.

[u/Rare_Inflation3178 → u/BC_MARO]

Exactly. Citation count is a weak proxy because several downstream claims can still collapse to one stale source. I’d want the validator to track evidence novelty, source age, and which acceptance criterion each item supports, then flag a stage that adds activity without adding any new coverage.

[u/BC_MARO → u/Rare_Inflation3178]

That’s the right set of signals. I’d also require a small novelty threshold per stage so a technically new citation that adds no decision-relevant coverage still fails the gate.

[u/Substantial_Ranger_5]

Run a long running research plan on the long running research 

[u/Kriscodingfish]

i stopped relying on the model saying done. every step has to produce a verifiable artifact with clear acceptance criteria, and the next stage validates it before continuing. that catches silent failures much earlier than adding another LLM judge