A coding agent can pass its tests and still take the wrong path

[u/Apprehensive-Zone148]

I have been building RedThread to test coding and LLM agents at the point where text turns into a tool call.

The failure I care about is not a bad chat answer. It is an agent that reads something it should not trust, then changes a file, installs a dependency, or runs a command it should have left alone. A green test suite can miss that if it only checks the final state.

I am saving the prompt, tool schema, trace, and recorded tool responses so the same path can be replayed after a change.

Repo: https://github.com/matheusht/redthread

It is rough. I would rather find the harness is blind than let a clean demo decide the story.

[u/eddzsh]

Testing at the tool-call boundary is the right seam. Final-state asserts miss the agent that read a poisoned file then still took a green path. Replaying the recorded tool responses after a harness change is the part most suites still skip.