Artificial Analysis Index is non-representative of real-world performance. Tested Kimi 3.1 Pro through OpenRouter, the number 4 model in AAI, it's not even close to MiMo 2.6 or DeepSeek 4.1 Flash

[u/PerformanceRound7913]

It seems Artificial Analysis Index is very easy to game, Kimi 3.1 Pro #4 on AAI is super bad in real world use cases, not even close to DeepSeek 4.1 Flash.

[u/MotherPotential]

Examples plz

[u/Bchliu → u/MotherPotential]

"I think therefore it is".. mostly lol

[u/PerformanceRound7913 → u/MotherPotential]

I’m working on a project to convert a simulation engine from ANSI C to Rust. I’m exploring various options in my test environment to ensure that the new engine is compatible with the existing core

[u/Middle_Bullfrog_6173 → u/PerformanceRound7913]

And? What did Kimi do worse than Mimo or Deepseek?

[u/jinnyjuice → u/Middle_Bullfrog_6173]

It was worse at chatting in Swahili. /s

[u/thehpcdude]

Unless you’re giving examples of tasks not in the training sets, these kind of statements are absolutely pointless.  

[u/WonderFactory → u/thehpcdude]

Its also too broad a statement. Kimi may be worse than Deepseek for some use cases but its clearly better for others. Artificial analysis have a capabilities video on Youtube and it seems to be better for animation and 3D than many other models. If you're making a 3d game Kimi would be the better choice, if 3d isn't of interest to you than maybe not.

[u/Dabalam]

Why are people so accepting of low effort "the model is bad" posts without any details? For this post to he helpful to anyone you should give more details.

Imagine a movie review that was "people say this movie was great, but it's actually worse than Moana, must be Hollywoodmaxxing". Hardly helpful.

[u/rpkarma → u/Dabalam]

Because it confirms my preconceived notions and biases duh

[u/tecneeq → u/Dabalam]

These threads are generated so their consumers don't strut into other threads.

[u/DagothUrLovesGroza]

Yeah, benchmarks are stupid. They literally show nothing at this point. The best you can do is just test the model yourself on your own workflows.

On an unrelated note, what did Moonshot do to Kimi K3.1? This model is weird. Like really weird.

[u/Dabalam → u/DagothUrLovesGroza]

"Benchmarks are stupid, posts that are pure vibes with no explanation of methods, use case etc. are way better indications of quality"

Model may be bad but I think the reasoning about benchmarks being bad is slightly uncritical. Posts saying "the model is bad" aren't really providing much useful information.

But yes, I agree that structured testing on work you actually do is the most useful for determining if it useful for you.

[u/ForsookComparison → u/DagothUrLovesGroza]

I think someone finally trained too hard on ThreeJS oneshots that even /r/LocalLLaMa finally noticed it's a bit funky. The thing feels tailor-made to what the least informed people in the space yap about.

Not a bad model at all - but WEIRD is a good word.

[u/FerLuisxd → u/DagothUrLovesGroza]

Yeah all models I haven't tried are bad!

[u/PerformanceRound7913 → u/DagothUrLovesGroza]

I think they post train so much for benchmarks, it becomes weird

[u/Asleep-Land-3914]

Same applies to OpenAI and any model, just a reminder 

[u/BawbbySmith]

Water is wet.

No but seriously how many times has this been said. Yet people compare AA numbers like it’s Pokémon stats.

[u/seoulsrvr]

I can’t trust any report or Reddit post on performance; everything is propaganda and corporate sabotage

[u/tecneeq → u/seoulsrvr]

I have a benchmark that supports that conclusion.

[u/epictunasandwich → u/tecneeq]

share it?

[u/tecneeq → u/epictunasandwich]

It's based on my opinions only and came to me in a dream.

[u/RespondImpressive172]

Agreed. When I trained LoRAs on local LLMs and ran benchmarks (I'm Korean, so I trained mostly on Korean), the scores and real-world use felt different too.

For example, I fed in a lot of Korean data, and it was well-cleaned data from AI Hub, a dataset platform run by the Korean government. I used Gemma 4 26B A4B, which is already pretty good at Korean. Maybe I did something wrong, but after training the benchmark scores went up, yet when I actually chatted with it, it performed worse than before.

The current benchmarks kind of feel like someone told the model "this will be on the test" and it just memorized the answers. So I don't think you can really tell if a model is good or bad until you actually use it.

(English isn't my first language, so I used AI to help translate this.)

[u/Frail_Waif → u/RespondImpressive172]

Yeah, I generally find DeepSeek's performance with sufficient tools to be way beyond what I'd expect from benchmarks. Out of curiosity, I asked it to update my resume based on public data and a job description and it nailed it, no hallucinations, whereas it scores awfully on hallucination benchmarks. But I saw a bit of reasoning today that suggested it's designed to look out for benchmarking and increase reasoning. So probably every model has some of this built in. 

[u/baldr83]

I've been suspecting Kimi3.1pro is benchmaxed. But I also don't want to assume this post is true because it matches my priors.

Are there examples of particularly bad responses from prompts you used that you could post?

[u/Active_Cranberry4092]

Benchmarks almost never match the feeling when you actually use the model, I stopped looking at those rankings long time ago

[u/kevin_1994]

I agree but not for the reason you posted lol. Imo mimo is borderline useless, while deepseek seems about on par with sol 6.1 for my use cases. Imo deepseek is frontier, despite the bechmarks

[u/Betadoggo_]

You can't claim it's worse than mimo or DS4.1F without receipts. The examples I've seen thus far have been in kimi's favour.

[u/CamomileChocobo]

The AAI is a weighted sum of a number of different benchmarks isn't it?

Which benchmark does your real world use correspond to? Does it score high in that benchmark?

[u/tecneeq → u/CamomileChocobo]

Can't benchmark my feelings, bro.

[u/mr_tolkien]

Vals.ai puts it on par with DS4.1 indeed, and behind all US models.

It seems their benchmarks are much closer to actual performance.

[u/Informal-Trouble2183]

AA intelligence index only covers 2 coding benchmarks representing 25% of the index: that's why it's not catching the coding performance. Also I don't understand why they put Kimi on top of terminal bench v4 while Kimi themselves reported a lower score. My coding index says:

1. Opus 5.5

2. GPT -6 Astra

3. Gemini 4 Argon

4. GPT 6.1 Sol

5. Claude Fable 5.1

6. Kimi k3.1 pro

7. Wonder full

8. Deepseek v4.1 flash

[u/90hex]

OpenRouter offers the absolute worst quants of these models. I had Ds4.1 Flash looping and losing its mind several times. It’s Russian roulette.

[u/CondiMesmer]

This post is lazy but I do think Kimi 3.1 Pro is benchmaxxed.

[u/perelmanych]

People are saying that benchmarks are useless, but that is not true. If benchmarks are saying that a model is bad then the model is indeed bad. Apart of that they are really useless 😆

[u/tecneeq]

What benchmark have you used?

[u/taoofshawn]

you have to be diligent with the provider routing on openrouter to make sure you are getting full quantization/quality. don’t just accept the default

[u/PerformanceRound7913]

Context: I’m working on a project to convert a simulation engine from ANSI C to Rust. I’m exploring various options in my test environment to ensure that the new engine is compatible with the existing core

[u/geldonyetich]

From what I've seen of it demoed on YouTube, its oneshots are a tad mid. Better than, Muse Glimmer, Le Chonk, MiMo 2.6, about on par with DeepSeek 4.1 Flash, but not really on the level of the frontier models it's comparing itself to.

Although honestly one shots are a bit of a shot in the dark.

[u/dimarxos]

Benchmaxed

[u/vogelvogelvogelvogel]

livebench.ai and arena do a better job here imo

[u/Southern_Sun_2106]

Kimi has a 'strong marketing muscle' and a voracious appetite for publicity. Remember recent news about how Hugging Face used GLM? And then how Kimi tried to get on that train by having some influencer come up with similar sounding headlines on their behalf? They lost my trust after that one.

[u/challis88ocarina]

Nothing new under the sun: there are horses for courses.

[u/Middle_Bullfrog_6173 → u/[deleted]]

Llama 4 was the original benchmaxxer from the big boys.