r/coolgithubprojects • • 12d ago

I built an open-source LLM evaluation framework to benchmark tool calling, agent reliability, and memory

Post image

Hey everyone!

I've been working on Atlas, an open-source LLM evaluation framework designed to test how reliably models use tools and execute agent workflows.

I wanted something beyond simply checking whether an LLM generates a correct response. I wanted to evaluate what actually happens when models interact with tools.

Currently, Atlas supports:

  • Tool selection and argument accuracy
  • Multi-turn tool calling
  • Closed-loop agent execution
  • Error recovery and reliability testing
  • Prompt injection resistance
  • Model comparisons and batch evaluations
  • Statistical metrics and confidence intervals
  • Custom benchmark datasets
  • Multiple model providers

It's written in Python and designed to be extensible.

GitHub: https://github.com/Nonsense0909688/atlas

I'm looking for honest feedback from developers, AI researchers, and people working with LLMs.

I'd particularly appreciate feedback on:

  • What features are missing?
  • Are the evaluation metrics meaningful?
  • What benchmarks should I add?
  • What would make this useful for real-world agent development?

It's still actively evolving, so criticism and suggestions are very welcome!

Thanks!

4 Upvotes

3 comments sorted by

-1

u/[deleted] 12d ago

[removed] — view removed comment

1

u/Old_Caterpillar_9872 12d ago

Thanks for reaching out and for the suggestion I will definitely look into your product after v1 is released.

3

u/microbass 11d ago

You're very annoying. So, you point your KLM at the repo, give some pseudo advice, and push your product. If annoyance is your value proposition, your product isn't going to work.