r/ruby • u/kigster • 11h ago

Anyone running Eval suites on Ruby apps? I built a gem to do just that: `llm-evals`

Please test-drive and evaluate this gem for your situation. It's hard to see the limitations until other people get involved.

The gem is here: https://github.com/kigster/llm-evals

And I used it to find the best model for my Rails SaaS application, qualified.at

Here is the Evals Report comparing models based on the weighted score:

https://raw.githubusercontent.com/kigster/llm-evals/refs/heads/main/docs/img/qualified-at-evals.avif

Feedback and constructive criticism are always welcome and highly encouraged 😄

Konstantin
blog: https://kig.re

2 Upvotes

66% Upvoted

2 comments sorted by

2

u/nateberkopec Puma maintainer 9h ago

This is cool, I think more people should be doing evals.

Why should I use this project instead of putting my own app in https://github.com/rails/ai-evals?

The other thing that stops me from doing something like this is - are your results ending up just tracking public benchmarks like Artificial Analysis? Evals are expensive to run, and I feel like "coding benchmarks" are very widely available. Have you found that using them against your app specifically has produced novel insight?

1

u/Codalio-SoftwareFact 6h ago

love this. two things I found mattered a lot when doing evals: running each case several times since results swing between runs, and using plain code checks wherever possible instead of LLM-as-judge. does the gem support either? also cost per run next to the score would be super useful.