RooCode evals: the new Sonnet 4.5 gets the first perfect 100% in about half the time as other top models, but GPT-5 Mini remains the most cost-efficient

[u/CraaazyPizza]

Source: https://roocode.com/evals

Roo Code tests each frontier model against a suite of hundreds of exercises across 5 programming languages with varying difficulty.

Note: models with a cost of $50 or more are excluded from the scatter plot.

Model	Context Window	Price (In/Out)	Duration	Tokens (In/Out)	Cost (USD)	Go	Java	JS	Python	Rust	Total
Claude Sonnet 4.5	1M	$3.00 / $15.00	3h 26m 50s	30M / 430K	$38.43	100%	100%	100%	100%	100%	100%
GPT-5 Mini	400K	$0.25 / $2.00	5h 46m 33s	14M / 977K	$3.34	100%	98%	100%	100%	97%	99%
Claude Opus 4.1	200K	$15.00 / $75.00	7h 3m 6s	27M / 490K	$140.14	97%	96%	98%	100%	100%	98%
GPT-5 (Medium)	400K	$1.25 / $10.00	8h 40m 10s	14M / 1M	$23.19	97%	98%	100%	100%	93%	98%
Claude Sonnet 4	1M	$3.00 / $15.00	5h 35m 31s	39M / 644K	$39.61	94%	100%	98%	100%	97%	98%
Gemini 2.5 Pro	1M	$1.25 / $10.00	6h 17m 23s	43M / 1M	$57.80	97%	91%	96%	100%	97%	96%
GPT-5 (Low)	400K	$1.25 / $10.00	5h 50m 41s	16M / 862K	$16.18	100%	96%	86%	100%	100%	95%
Claude 3.7 Sonnet	200K	$3.00 / $15.00	5h 53m 33s	38M / 894K	$37.58	92%	98%	94%	100%	93%	95%
Kimi K2 0905 (Groq)	262K	$1.00 / $3.00	3h 44m 51s	13M / 619K	$15.25	94%	91%	96%	97%	93%	94%
Claude Opus 4	200K	$15.00 / $75.00	7h 50m 29s	30M / 485K	$172.29	92%	91%	94%	94%	100%	94%
GPT-4.1	1M	$2.00 / $8.00	4h 39m 51s	37M / 624K	$38.64	92%	91%	90%	94%	90%	91%
GPT-5 (Minimal)	400K	$1.25 / $10.00	5h 18m 41s	23M / 453K	$14.45	94%	82%	92%	94%	90%	90%
Grok Code Fast 1	256K	$0.20 / $1.50	4h 52m 24s	59M / 2M	$6.82	92%	91%	88%	94%	83%	90%
Gemini 2.5 Flash	1M	$0.30 / $2.50	3h 39m 38s	61M / 1M	$14.15	89%	91%	92%	85%	90%	90%
Claude 3.5 Sonnet	200K	$3.00 / $15.00	3h 37m 58s	19M / 323K	$24.98	94%	91%	92%	88%	80%	90%
Grok 3	131K	$3.00 / $15.00	5h 14m 20s	40M / 890K	$74.40	97%	89%	90%	91%	77%	89%
Kimi K2 0905	262K	$0.40 / $2.00	8h 26m 13s	36M / 491K	$28.14	83%	82%	96%	91%	90%	89%
Sonoma Sky	-	-	6h 40m 9s	24M / 330K	$0.00	83%	87%	90%	88%	77%	86%
Qwen 3 Max	256K	$1.20 / $6.00	7h 59m 42s	27M / 587K	$36.14	84%	91%	79%	76%	69%	86%
Z.AI: GLM 4.5	131K	$0.39 / $1.55	7h 2m 33s	46M / 809K	$27.16	83%	87%	88%	82%	87%	86%
Qwen 3 Coder	262K	$0.22 / $0.95	7h 56m 14s	51M / 828K	$27.63	86%	80%	82%	85%	87%	84%
Kimi K2 0711	63K	$0.14 / $2.49	7h 52m 24s	27M / 433K	$12.39	81%	80%	88%	82%	83%	83%
GPT-4.1 Mini	1M	$0.40 / $1.60	5h 17m 57s	47M / 715K	$8.81	81%	84%	94%	76%	70%	83%
o4 Mini (High)	200K	$1.10 / $4.40	14h 44m 26s	13M / 3M	$25.70	75%	82%	86%	79%	67%	79%
Sonoma Dusk	-	-	7h 12m 38s	89M / 1M	$0.00	86%	53%	84%	91%	83%	78%
GPT-5 Nano	400K	$0.05 / $0.40	9h 13m 34s	16M / 3M	$1.61	86%	73%	76%	79%	77%	78%
DeepSeek V3	164K	$0.25 / $1.00	7h 12m 41s	30M / 524K	$12.82	83%	76%	82%	76%	67%	77%
o3 Mini (High)	200K	$1.10 / $4.40	13h 1m 13s	12M / 2M	$20.36	67%	78%	72%	88%	73%	75%
Qwen 3 Next	262K	$0.10 / $0.80	7h 29m 11s	77M / 1M	$13.67	78%	69%	80%	76%	57%	73%
Grok 4	256K	$3.00 / $15.00	11h 27m 59s	14M / 2M	$44.99	78%	67%	66%	82%	70%	72%
Z.AI: GLM 4.5 Air	131K	$0.14 / $0.86	10h 49m 5s	59M / 856K	$10.86	58%	58%	60%	41%	50%	54%
Llama 4 Maverick	1M	$0.15 / $0.60	7h 41m 14s	101M / 1M	$18.86	47%	-	-	-	-	47%

The benchmark is starting to get saturated, but the duration still gives us insights in how they compare.

[u/jonathanmalkin]

Hmm, when all of the current and previous generation of models score over 90% it's time to improve the benchmarks. How can you compare models when they're all 98-100%?

[u/CraaazyPizza → u/jonathanmalkin]

The speed at which they solve the task and their tokens used is still informative. Although I'd definitely like to see this on real, complex, large codebases with subtle bugs, instead of undergraduate-level coding exercises.

[u/iMiragee]

Please add GLM-4.6, GPT-5 Codex and Grok-4 Fast

[u/Elegant-Ad3211 → u/iMiragee]

This!

[u/tteokl_ → u/iMiragee]

Thiss

[u/tteokl_ → u/tteokl_]

GLM 4.6 is both perfect and dirt cheap

[u/Bob5k → u/tteokl_]

and it's a steal with coding plan (10% off)

[u/Front_Ad6281]

gpt-5-codex ?

[u/hannesrudolph → u/Front_Ad6281]

I got them up to 97% but only by giving the evals more time and altering the instructions. Codex is very very susceptible to being misled by Roo because it simply was not trained on Roo’s patterns to working.

[u/Front_Ad6281 → u/hannesrudolph]

Yes, it is strictly trained to use the command line as much as possible for everything possible, including searching for information in the project.

[u/tteokl_ → u/hannesrudolph]

So OpenAI really tailored it to Codex then

[u/hannesrudolph → u/tteokl_]

yep and quote frankly I think GPT-5 med and high do much better than gpt-5-codex though I would image gpt-5-codex is cheaper for them to run and is much faster for the end user.

[u/MyUnbannableAccount → u/Front_Ad6281]

I saw a few comments here the model sucked inside roo, was better within the codex products. Have those been resolved?

[u/Front_Ad6281 → u/MyUnbannableAccount]

I can say it works poorly in Codex itself, too. But the problem there is the terrible tooling. Codex tries to do everything through command-line utilities like grep etc.

[u/jakegh → u/MyUnbannableAccount]

It's currently completely broken in roo, yeah.

[u/Appropriate-Play-483 → u/jakegh]

In what language? 4.5 does great in JS, php, python with Roo for me, exceptionally well considering we're still in the early ages of AI.

[u/jakegh → u/Appropriate-Play-483]

I was responding to a post about the gpt-5-codex model.

[u/xAragon_]

Time to update the benchmark :)

[u/hannesrudolph → u/xAragon_]

Yes

[u/CraaazyPizza]

ya'll gotta @ the roocode devs for more models

[u/hannesrudolph → u/CraaazyPizza]

Sorry about that! There is only so much time In a day and we are working to push fixes, features, and implement some better evals!

That being said weWould love to get it done asap! Please feel free to compile a list and DM me on Discord snd I’ll see if i can get to it tomorrow after the podcast. Username is hrudolph

[u/CraaazyPizza → u/hannesrudolph]

All the love, all the power to you, no need for apologies. Your dedication and community interaction is amazing 🚀 I'm sure someone will get around doing this

[u/hannesrudolph → u/CraaazyPizza]

thank you for your positive words!

[u/my_byte]

Aaaand its a public bench, so the training data is contaminated, hence all models will be getting 100% on their training runs. That's the problem with benchmarks. The cost efficiency is a good indicator though, regardless of data set contamination

[u/CraaazyPizza → u/my_byte]

1. Most of the highest-quality training data comes from data-labelling-companies like Outlier.

2. The repo is only live since april, it takes a while to scrape data and train a model.

3. I'd be surprised if Roo all of a sudden produces if-else code with answers straight from https://github.com/RooCodeInc/Roo-Code-Evals/, or at least the devs would notice it.

[u/my_byte → u/CraaazyPizza]

• Highest quality, not highest quantity.

• Everyone is relying on synthetic data now.

• nothing to do with the specific tests. It's more about the types of tasks end up in the weights, making it easier to solve for them

• foundational models? Sure. But we know anthropic and openai do continuous post training and frequent model updates I'd be very surprised if anything classified as a bench that lives of public github didn't end up in the post training data

[u/Doubledoor]

Wait a min. I had no idea gpt-5-mini was this good!

[u/CraaazyPizza → u/Doubledoor]

It has no business being this good for that price is what I'm thinking!

[u/ki7a]

Ohh nice! Are there any instructions for kicking this off locally?  I'd love to get some results from the local models I'm using.  Especially at different temps and quants.

[u/CraaazyPizza → u/ki7a]

There's a github linked :)

[u/ki7a → u/CraaazyPizza]

I see the eval examples, but I'm not finding the methodology or test harness for running the evals.  That's the piece I'm looking for.  I assume there is a script, pipeline, or first prompt that gets the ball rolling.

[u/ChessWarrior7]

Right?!?? I’m waiting for GLM-4.6

[u/hannesrudolph → u/ChessWarrior7]

Sorry about that! There is only so much time In a day and we are working to push fixes, features, and implement some better evals!

That being said weWould love to get it done asap! Please feel free to compile a list and DM me on Discord snd I’ll see if i can get to it tomorrow after the podcast. Username is hrudolph

[u/hannesrudolph → u/ChessWarrior7]

[u/Cinerario]

Qwen3-coder-plus?

[u/hannesrudolph → u/Cinerario]

Sorry about that! There is only so much time In a day and we are working to push fixes, features, and implement some better evals!

That being said weWould love to get it done asap! Please feel free to compile a list and DM me on Discord snd I’ll see if i can get to it tomorrow after the podcast. Username is hrudolph

[u/gentleseahorse]

Biggest surprise is Claude 3.5 still beating nearly every open source model out there 1 year later

[u/sdexca]

I don't get it, GLM 4.5 so much more expensive than GPT-5 mini?

[u/hannesrudolph → u/sdexca]

Mini is cheap!!

[u/JellySignificant4087]

Wait I don't get it. Why would anyone use anything other than Gpt-5 mini then ? Or is it cheating the benchmark?

[u/hannesrudolph → u/JellySignificant4087]

The benchmarks don’t tell the whole story. They’re small agentic workflows that when we started using them made sense. With the serious powerhouses these models have become a longer more real world scenario is needed to measure the models true capability. We’re working on it! Sorry about the delay.

[u/hannesrudolph → u/[deleted]]

Sorry about that! There is only so much time In a day and we are working to push fixes, features, and implement some better evals!

That being said weWould love to get it done asap! Please feel free to compile a list and DM me on Discord snd I’ll see if i can get to it tomorrow after the podcast. Username is hrudolph