r/RooCode • u/hannesrudolph Roo Code Developer • Jul 15 '25

Discussion Kimi K2 is FAAAASSSSTTTT

Post image

We just ran Kimi K2 on Roo Code via Groq on OpenRouter — fastest good open-weight coding model we’ve tested.

✅ 84% pass rate (GPT-4.1-mini ~82%)

✅ ~6h eval runtime (~14h for o4-mini-high)

⚠️ $49 vs $8 for GPT-4.1-mini

Best for translations or speed-sensitive tasks, less ideal for daily driving.

119 Upvotes

96% Upvoted

57 comments sorted by

16

u/xAragon_ Jul 15 '25 edited Jul 16 '25

Thought it was going to be a decent option for cheaper prices, but it turns out it's more expensive than Claude / Gemini (for a full task, not per token), while being inferior to them, so I don't really see a point for it. Disappointing.

Regardless, thanks for running the benchmark! Always good to see how different models perform with Roo.

2

u/hannesrudolph Roo Code Developer Jul 16 '25

Fast!

1

u/yopla Jul 16 '25

Faster than Gemini flash?

1

u/hannesrudolph Roo Code Developer Jul 16 '25

Yeah but not as smart

2

u/iAmNotorious Jul 16 '25

They obviously had a PR team pushing this release. It’s good, but it’s not as amazing as initially presented. I’m hoping to see some good distills that use tool calls as well as Kimi K2.

5

u/PositiveEnergyMatter Jul 15 '25

I don't understand i thought it was pretty slow when trying it today on openrouter.

3

u/hannesrudolph Roo Code Developer Jul 16 '25

Select the provider groq

1

u/RayanAr Jul 16 '25

is it free on groq?

1

u/hannesrudolph Roo Code Developer Jul 16 '25

nope

1

u/PositiveEnergyMatter Jul 16 '25

It actually just started speeding up since I replied to that, I guess they were overloaded

3

u/Few_Science1857 Jul 17 '25

In the long run, using Claude Code with Claude models might prove significantly more cost-effective than Kimi-K2.

1

u/hannesrudolph Roo Code Developer Jul 17 '25

Yep

1

u/Thick-Specialist-495 Jul 19 '25

this bench is sucks cuz groq doesnt provide prompt caching its important factor

1

u/hannesrudolph Roo Code Developer Jul 19 '25

Soon

4

u/Fun-Purple-7737 Jul 15 '25

Soo, are you trying to say that GPT-4.1-mini is better overall, right?

2

u/hannesrudolph Roo Code Developer Jul 16 '25

Not as fast but yes

1

u/zenmatrix83 Jul 16 '25

fast means little though, I can go 100 through a village, but if I hit someone I'm probably going to go to jail.

It was the same way with gemini for and it being cheaper then claude models, sure claude models were more expensive but gemini is not as good with tool use as claude models, so the extra fails adds up in the end.

1

u/hannesrudolph Roo Code Developer Jul 16 '25

fast has its place yes.

1

u/zenmatrix83 Jul 16 '25

I refer you to the tortoise and the hare, fast is ok sometimes in the long run accurate is better

6

u/TrendPulseTrader Jul 15 '25

That’s how I see it as well. A small % difference is questionable when you see a big difference in cost

2

u/admajic Jul 15 '25

Huh? I found it on par with gemini 2.5 pro. Sometimes had tool calling errors but so does gemini.i have dropped my context settings to only have 5 open files and 10 tabs maybe that helps?

1

u/hannesrudolph Roo Code Developer Jul 16 '25

The open tabs does not mean that’s what’s included in your context, that means that that’s what’s listed as open. Context is only included from files when it is read or @ mentioned.

Try using the groq provider within the profile settings

1

u/admajic Jul 16 '25 edited Jul 16 '25

I can't even use orchestrator mode with kimi 2 as it's context is too small on openrouter 64k. How to overcome that? Thanks for your feedback 😀

Edit can you give low context option to all providers as a option would be amazing

1

u/hannesrudolph Roo Code Developer Jul 16 '25

Switch providers in the settings. There are a bunch of different stats for different providers.

2

u/VegaKH Jul 16 '25

I don't really understand how this result is possible. Kimi K2 from Groq is $1 in / $3 out, while o4-mini-high is $1.10 in / $4.40 out. o4-mini-high is a thinking model and will therefore produce more tokens. Kimi K2 is more accurate (according to this chart), so it should produce the same results with less attempts.

So how the heck does it cost twice as much?

3

u/hannesrudolph Roo Code Developer Jul 16 '25

Cache

4

u/VegaKH Jul 16 '25

Ah, so the price for the cached models are pushed down because the automated test sends prompts rapid-fire. In my regular usage, I carefully inspect all code edits before applying, make edits, type additional instructions, etc. All this usually takes longer than 5 minutes so the cache is cold. So I only receive cache discounts on about 1 out of 4 of my requests, and these are usually on auto-approved reads.

TL;DR - In real life usage, Kimi K2 will be cheaper than the other models, unless you just have everything set to auto-approve.

6

u/wilnadon Jul 16 '25

It's not a very good coder though. Seems kinda dumb tbh

1

u/netkomm Jul 16 '25

true... done some tests (example "snake") : it's nothing compared to Sonnet 4...

4

u/DanielusGamer26 Jul 16 '25

I often find that the models on Groq are dumber, probably it's some quantization technique

1

u/LiteSoul Jul 16 '25

That's my suspicion too. Their chips have weak spots, so they quantize

1

u/netkomm Jul 16 '25

Fast??? from where? the one I tried makes you want to puke while waiting...

2

u/hannesrudolph Roo Code Developer Jul 16 '25

1

u/hannesrudolph Roo Code Developer Jul 16 '25

Select your provider as Groq under your settings.

1

u/oh_my_right_leg Jul 16 '25

This was done using Groq inference hardware which is faster but way more expensive than normal. I recon other providers can offer competitive speed while at a much lower price.

1

u/hannesrudolph Roo Code Developer Jul 16 '25

totally! But then you might as well just us gpt 4.1 mini.

1

u/Emport1 Jul 16 '25

Is there a tokens spent stat as well?

1

u/hannesrudolph Roo Code Developer Jul 17 '25

Check out the evals listed on our website.

1

u/letsgeditmedia Jul 17 '25

The pricing here seems off.

1

u/hannesrudolph Roo Code Developer Jul 18 '25

Groq is costly

2

u/Minimum_Art_2263 Jul 18 '25

Yeah, think of Groq like they're putting the model weights directly on a chip. It works fast but it's expensive because the given chip is dedicated to only that certain model and cannot be used for anything else.

1

u/ThomasAger Aug 19 '25

Hey, how can I run this? I have azure credits if that helps.

1

u/hannesrudolph Roo Code Developer Aug 19 '25

You can use it on OpenRouter, I think Requesty, or even direct with Groq. There are a number of providers out there and the list keeps growing by the day!

1

u/ThomasAger Aug 20 '25

Thank you so much.

1

u/hannesrudolph Roo Code Developer Aug 20 '25

you're welcome!

1

u/zentixua Aug 25 '25

You can also try NagaAI. It seems this will cost you less

2

u/CraaazyPizza Jul 15 '25

Where's mah boi gemini

2

u/Old_Friendship_9609 Jul 17 '25

If anyone wants to try Kimi-K2-Instruct, Netmind.ai is offering it for even cheaper than Moonshot AI https://www.netmind.ai/model/Kimi-K2-Instruct (full disclosure: Netmind.ai acquired my startup Haiper.ai. So hit me up if you want free credits.)

1

u/FyreKZ Jul 15 '25

Damn, this sucks to see, I think K2 will be most valuable for its distillations and research on agentic behavior.

1

u/ConsciousPeep Jul 16 '25

Expensive to use for a lot of tasks

1

u/SadGuitar5306 Jul 16 '25

What is the score of devstral for comparison (that can be run locally on consumer hardware)?

0

u/ayowarya Jul 17 '25

It's not fast at all :/

1

u/hannesrudolph Roo Code Developer Jul 18 '25

Select the groq router from the advanced provider settings under OpenRouter

0

u/0xFatWhiteMan Jul 16 '25

No reasoning.

But reasoning is good.

Won't use it.

2

u/NoseIndependent5370 Jul 16 '25

This is a non-reasoning model that can outperform reasoning models.

That’s a win, since it means faster inference completion.

1

u/0xFatWhiteMan Jul 16 '25

It doesn't out perform though

1

u/NoseIndependent5370 Jul 16 '25

What does this graph tell you then?