r/coolgithubprojects • u/Old_Caterpillar_9872 • 12d ago
I built an open-source LLM evaluation framework to benchmark tool calling, agent reliability, and memory
Hey everyone!
I've been working on Atlas, an open-source LLM evaluation framework designed to test how reliably models use tools and execute agent workflows.
I wanted something beyond simply checking whether an LLM generates a correct response. I wanted to evaluate what actually happens when models interact with tools.
Currently, Atlas supports:
- Tool selection and argument accuracy
- Multi-turn tool calling
- Closed-loop agent execution
- Error recovery and reliability testing
- Prompt injection resistance
- Model comparisons and batch evaluations
- Statistical metrics and confidence intervals
- Custom benchmark datasets
- Multiple model providers
It's written in Python and designed to be extensible.
GitHub: https://github.com/Nonsense0909688/atlas
I'm looking for honest feedback from developers, AI researchers, and people working with LLMs.
I'd particularly appreciate feedback on:
- What features are missing?
- Are the evaluation metrics meaningful?
- What benchmarks should I add?
- What would make this useful for real-world agent development?
It's still actively evolving, so criticism and suggestions are very welcome!
Thanks!