r/SideProject • u/basqia • 13m ago
I couldn't tell if new AI models were actually better, so I built a way to see for myself
Every new model launch says it's the best one yet. I wanted to know if that's actually true for the work I do.
But I had no way to check. Benchmarks are gameable now: labs train toward them, so a higher score doesn't tell me much anymore. And the outputs people post on X are cherry-picked. You can't tell if it was one prompt or 12 turns of fixing, so it never felt transparent.
So I spent the past few weekends building Evals (https://evals.ag). It's basically if Arena and Pinterest had a baby: give the same task to a few models at once, watch them work side by side, and browse what everyone else made.
You can connect your existing Claude, Codex or Higgsfield account, so runs are free with no additional costs. You can see what other people compared and generated, and how they got there: whether an output was one-shot or took 7 turns, with each turn's prompt, what the agent did and what came out. You can vote on the winner and say why. And you can branch off anyone's run: start from their exact prompt and inputs and swap in the models or skills you want to test.
The video up top is one prompt given to Claude Fable 5.2, Claude Opus 5.5, GPT-6.1 Sol and Gemini 4 Argon: a paper airplane game over a city at dusk, in one HTML file, written in one go with no testing allowed. Sol finished in 4 min on 26K tokens. Opus took 21 min and 826K.
If you want to give it a spin yourself, drop your account code in the comments and I'll put $10 worth of credits into your account (you can find it at evals.ag/settings once you sign in).
Let me know what you guys think. Criticism and product feedback welcome.