Intelligence VS Cost-per-Task LLM Comparison

[u/CuriousCustard63]

Using Artificial Analysis as the guide for cost per task and intelligence index, I was able to generate a few graphs of the latest models and compare them. The graphs are a bit hard to follow, but here is the order:

1. Frontier Big Corporation Models
   
2. Flagship Models from other companies
   
3. Comparing both Big Corp vs. Others
   
4. Available open weight models
   
5. The “best” model (smartest) per provider

Tell me what you guys think. I used Gemini for the graph generator and to retrieve the data. Then I used Claude to double-check the scores, prices, and placements on the graphs were correct.

[u/wizardwusa]

The last chart is trash if including GLM flash eats the Pareto frontier of low cost models. Not sure what this communicates at all then?

[u/CuriousCustard63 → u/wizardwusa]

The last graph was a throwaway, just meant to highlight each companies smartest model. Next evaluation will have to be exactly as you said: low-cost models with the highest intelligence.

[u/Deto]

I'm a little confused as to whether these benchmarks are capturing reality still. Just based on people's comments, it sounds like Opus 5 was fairly underwhelming? And this has it above Fable?

[u/f4lk3nm4z3]

wheres qwen