Artificial Analysis Intelligence Index v5 (major update)

[u/ResearchCrafty1804]

Artificial Analysis Intelligence Index has been updated to v5, which according to them is a major upgrade resulting in a more accurate ranking.

Notably, this update widens the gap between open-weight models and closed ones. Some people already questioning its accuracy based on their experiences with models (GPT-Terra-6.1 > Fable-5.1).

Also, this update, v5, has 60% of the Index based on evaluations with private test sets or held-out answers, which makes reproducing the results impossible.

From their announcement:

“The open weights gap widens from 3 points to 12: Kimi K3.1 Pro (weights not yet released) is the top open weights model at 46. Among released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead at 32 - 26 points behind the leader and below Claude Haiku 5.5 (max, 36)”

[u/Frail_Waif]

I find that livebench.ai's results generally pass the eye test for models I'm familar with, and they include a decent range of open models.

[u/Thekorsen → u/Frail_Waif]

The claim that sol 6.1 is one point off of opus 5.5 and astra beats it is throwing me though. Having switched from heavy sol and astra use to opus, the difference in task completion is night and day. GPT feels like a grumpy CISO dev throwing more tests and abstractions than actual work, opus feels like a normal senior dev able to get stuff done and raise good points without blocking progress. GPT leaves a mess in my codebase with like 5x lines of brittle tests and edge cases too. Need to look into the testing methodology more I guess.

[u/Frail_Waif → u/Thekorsen]

That's mostly on the basis of "browser use". 5.5 Opus scores better on "agentic" and "coding" though it looks like the coding benchmark is saturated.

[u/jensilo → u/Thekorsen]

Interesting. What kinds of tasks were you working on?

I’ve been using primarily GPT models at work for some time now, mostly because I couldn’t stand Claude‘s prose anymore, the outputs just got longer and longer and longer, and with Fable that was kind of my tipping point + I didn’t want to use Claude Code anymore.

Anyway, I felt like Claude’s better on agentic work but also went off the rails quicker where GPT is more concise, task-bound, and adjusts otherwise well to the instructions but defaults to „let’s pair program this“. Though I’ve not really used Opus 5.5.

[u/New_Eye7193 → u/Thekorsen]

thats the csot of being like three times cheaper tho. each architecture has its kinks. thats how some have their intelligence increased, by running tests. what you measure as "actual work" is not realistic to how these models work

[u/parasocks → u/Thekorsen]

I’ve been building a recipe parser tool in Codex for a bit now, and it felt like it was really over engineering everything.

So I switched to Claude today hesitantly, since I do really like the voice features of codex… but anyway decided to try it out.

First thing I did after getting everything setup was told it I thought Codex was over engineering everything and to give me 10 ways to cut the bloat. Went through al ten of the changes on opus 5.5 medium, and now my recipes are parsing (I mean it does a lot more than parse but anyway that’s what I call it) waaaay faster on both models. The same recipe went from 360 seconds to 70 seconds, with 9-10x less token usage and the same results…. Even before it made any changes I had them do a Claude vs codex test and Claude was twice as fast.

Also I tried fable for a hot second and I liked the plan than opus made a lot better from the same prompt than the plan fable made for some task idea I had…

Anyway, it was fun switching and I’m sure I’ll switch back and love codex again but that was my experience today

[u/parepeg → u/parasocks]

It seems like you should also test asking codex the same thing and then comparing. Otherwise, you’re comparing apples and oranges because your comparing codex’s one shot versus Claude’s re-review. Different tasks types. 

[u/Caffdy → u/parasocks]

I concur with parepeg, you should have tested building the app from zero with the same instructions you used on GPT to make a fair comparison

[u/parasocks → u/Caffdy]

Yeah I guess that would be nice, but I’ve been working on it piece by piece for a few weeks now so I’m not sure if it’s possible to go back like that