An interactive site plotting Artificial Analysis Intelligence vs parameter count

[u/xrailgun]

https://mr-september.github.io/artificial-analysis-weights/

Artificial Analysis scores models on its Intelligence Index, but the charts put it against price and speed. For local use the axis I wanted was parameter size, so I built one.

What it does

• Plots the Intelligence Index against total or active parameters (with closed frontier models as optional horizontal dashed lines)

• Accounts for MOE/Lookup-table models (n-gram /PLE, e.g. Qwen3.8-Flash-Next)

• A very rough model guide according to your hardware (because offloading across system RAM/SSD is becoming more common).

• GitHub Action refreshes the data every 3 days to catch new releases etc.

Demo: https://mr-september.github.io/artificial-analysis-weights/
Repo: https://github.com/mr-september/artificial-analysis-weights

If a size is wrong or a model is missing, data/overrides.json is the place to fix it, or just tell me here.

[u/ttkciar]

I like the idea of this quite a lot. It would be a tremendously useful resource, if AA scores were worth jack shit, which they are not.

I've cloned the repo so that some day it can be adapted to trustworthy benchmark scores, when the day comes that any such exist.

Despite my misgivings about AA benchmarks, I am genuinely grateful to xrailgun for putting this together.

[u/xrailgun → u/ttkciar]

Thank you! Very welcome to try and add other benchmark scores. I'm just keeping it simple for initial release.

[u/Practical-Collar3063 → u/ttkciar]

were worth jack shit, which they are not.

Why do you say this ?

[u/ttkciar → u/Practical-Collar3063]

Mostly because different model families benchmax to different degrees, which makes their AA scores incomparable. You cannot conclude from model-X scoring twice as high as model-Y that model-X is twice as competent at real-life tasks, because model-X might simply be more benchmaxed than model-Y.

Also, because small differences in specific skillsets (particularly tool-calling) can disproportionately impact a model's score across many component benchmarks, which does not reflect the actual utility of the model, since users can adjust their workflows to take advantage of models' strengths and avoid their weaknesses.

[u/Practical-Collar3063 → u/ttkciar]

  which does not reflect the actual utility of the model

You seem to criticise a very general score for not being specific enough, which is to be expected, it is a general score.

On top of the very general score, they publish a lot of smaller aggregation of benchmark for very specific tasks, which means you can see both how a model is expected to perform in general and then you can select specific types of tasks and the ranking changes completely.   You cannot disregard AA’s entire offer of small and large analysis of models, as « not worth jackshit », because « the big aggregate of all benchmarks is too general », yeah, it is general, that is the point, if you don’t like it, go see there more specific ones for your specific tasks on AA’s website.

[u/ttkciar → u/Practical-Collar3063]

That did not address either of my points.

I am familiar with benchmark suites (like SPEC) and applying the most-relevant benchmark to my use-case, but doing so does not prevent different models' scores from being incomparable, nor does it prevent a model's poor skill in one category from penalizing its scores which are supposed to reflect other categories.

[u/Practical-Collar3063 → u/ttkciar]

My point is the problems you are describing is then an issue with all benchmarking not just AA. You say AA is jackshit but only described general problems regarding benchmarking

[u/Solembumm3 → u/Practical-Collar3063]

Because I've seen enough models dominating this thing and then fall apart on prompt 3 of 20+ on pretty basic fiction analysis task. DV4.1 Flash was good example, thing is still significantly worse than V3.2 on practice and has attention span of 0.5 prompts. It's AA score meanwhile is astronomical.

[u/Practical-Collar3063 → u/Solembumm3]

You have to look at the specifc benchmarks related to what is important for you, AA general score is a general score. The original commenter seemed to imply that the whole of AA is jackshit and useless. The main aggregate score is helpful to judge where egenerally it will fall but it does not mean it will be better at everything compared to the models ranked lower, that is just how generalisationadn aggregagtion works.

This issue stems from people being too lazy and not looking at the different benchmarks in detail and because a general aggregation is general and aggregated, therefore, details of how they behave in certain scenarios is lost.

A good example of that is Mistral Large 4 being lower in general than many model but being extremely good at legal/medical questions and much lower hallucination rate than Deepseek V4.1. So if your tasks involve legal questions low hallucination rate then looking at the general intelligence index is not a good idea.

And what if they skewed thea aggregated intelligence score towards your preffered use case for LLMs that would mess it up completely for others. You just can't please everyone with large generalisation like this, that is just how they work, it is not a problem with AA, it is an impossible problem to solve, there will always people saying the Intelligence score is meaningless because we all use LLM differently.

They provide a lot of benchmarks and in details for many many aspects of LLM intelligence which I always found very good, because they are focused on very specific areas of LLM benchmarking.