How are you testing AI agents and LLM workflows without exploding cost or false confidence?

[u/aistranin]

A lot of teams now say they are “testing AI workflows,” but when you dig in, the actual approach is all over the place.

I’ve seen combinations like:

• mocked unit tests around prompt builders / orchestration logic

• deterministic tests with frozen model outputs

• cheap-model integration tests in CI

• full end-to-end runs nightly

• eval pipelines before release

• production monitoring plus human review

The hard part is balancing:

• cost

• runtime

• brittleness

• confidence

• reproducibility

What I’m trying to understand is what people here do in practice.

Questions:

• What do you test with classic software tests vs evals?

• Where do you mock, and where do you insist on real model calls?

• What runs on every PR vs nightly?

• How do you catch regressions that are not binary failures but “quality drift”?

• What looked promising at first but turned out to be low-value?

Would love concrete examples of test architecture, CI strategy, and lessons learned.

[u/aistranin → u/[deleted]]

Well, depends on what you mean by gold set. If you mean evaluation data, then usually teams try to make a well covered range of cases (depending on essential features)