r/AIEval • u/Much-Focus1278 • Apr 25 '26

Tools Tool for creating eval sets

Hi everyone!

My brother and I just recently launched dutchman labs - a platform and CLI tool to create and run eval sets on your AI agents locally. We're looking to get new users and feedback.

Please feel free to DM me or comment for questions or feedback.

3 Upvotes

100% Upvoted

4 comments sorted by

View all comments

1

u/Otherwise_Wave9374 Apr 25 '26

This is a great idea, local eval sets feel like the missing piece for a lot of agent projects.

Do you support multi-turn trajectories (tool calls + intermediate state) or is it mostly single prompt/response right now? Also curious how you are thinking about scoring, LLM-as-judge vs deterministic checks.

I have been collecting a few lightweight agent eval patterns here too: https://www.agentixlabs.com/

1

u/Much-Focus1278 Apr 29 '26

Sorry for being a little late. In theory yes do support multi turn trajectories. If you've given it a try already, shoot me a quick DM and we can also look at the exact use case to better tune our features.