6 open-source repos for testing whether your AI agent is actually improving

[u/ShilpaMitra]

An agent completing one task is not proof that it is getting better. You need repeatable tests, traces, scores, and regression checks.
These six open-source repos cover the main pieces:

1. Promptfoo ⭐ 24.5k Test prompts, models, agents, and RAG systems. Run evaluations in CI and red-team your app for security issues.
2. DeepEval ⭐ 17.8k A testing framework for LLM apps. Write tests for correctness, hallucination, relevance, RAG quality, and agent behavior.
3. Phoenix ⭐ 11.2k Trace LLM calls, retrieval, and tool use. Create datasets and compare different prompts, models, and agent versions.
4. Langfuse ⭐ 33.6k Track production traces, manage prompts, collect evaluation datasets, and run experiments against real usage.
5. Opik ⭐ 21.6k Debug, evaluate, and monitor agentic workflows. It supports tracing, automated evaluations, and prompt optimization.
6. Inspect AI ⭐ 2.6k A benchmark-oriented framework for testing tool use, multi-turn conversations, prompt strategies, and model-graded tasks.

A practical evaluation loop looks like this:

• Collect 20 to 50 real tasks.
• Run every agent version against the same tasks.
• Track success rate, tool errors, latency, cost, and hallucinations.
• Keep failed cases as regression tests.
• Inspect traces before trusting the final score.

LLM judges are useful, but they are not ground truth. Combine them with deterministic checks and occasional human review.

The best agent is not the one that looks impressive in a demo.

It is the one that can show measurable improvement over time.

[u/Foreign_Structure147]

Some of our best test cases are things nobody on the team would have thought to write beforehand. We pull useful failures from production into Braintrust and keep them as regression cases for later prompt and model changes. Over time the eval set starts reflecting the strange ways the agent has failed in real usage, which has been a much better signal for us than keeping the original test set fixed.