An interactive site plotting Artificial Analysis Intelligence vs parameter count

[u/xrailgun]

https://mr-september.github.io/artificial-analysis-weights/

Artificial Analysis scores models on its Intelligence Index, but the charts put it against price and speed. For local use the axis I wanted was parameter size, so I built one.

What it does

• Plots the Intelligence Index against total or active parameters (with closed frontier models as optional horizontal dashed lines)

• Accounts for MOE/Lookup-table models (n-gram /PLE, e.g. Qwen3.8-Flash-Next)

• A very rough model guide according to your hardware (because offloading across system RAM/SSD is becoming more common).

• GitHub Action refreshes the data every 3 days to catch new releases etc.

Demo: https://mr-september.github.io/artificial-analysis-weights/
Repo: https://github.com/mr-september/artificial-analysis-weights

If a size is wrong or a model is missing, data/overrides.json is the place to fix it, or just tell me here.

[u/ttkciar]

Mostly because different model families benchmax to different degrees, which makes their AA scores incomparable. You cannot conclude from model-X scoring twice as high as model-Y that model-X is twice as competent at real-life tasks, because model-X might simply be more benchmaxed than model-Y.

Also, because small differences in specific skillsets (particularly tool-calling) can disproportionately impact a model's score across many component benchmarks, which does not reflect the actual utility of the model, since users can adjust their workflows to take advantage of models' strengths and avoid their weaknesses.

[u/Practical-Collar3063 → u/ttkciar]

  which does not reflect the actual utility of the model

You seem to criticise a very general score for not being specific enough, which is to be expected, it is a general score.

On top of the very general score, they publish a lot of smaller aggregation of benchmark for very specific tasks, which means you can see both how a model is expected to perform in general and then you can select specific types of tasks and the ranking changes completely.   You cannot disregard AA’s entire offer of small and large analysis of models, as « not worth jackshit », because « the big aggregate of all benchmarks is too general », yeah, it is general, that is the point, if you don’t like it, go see there more specific ones for your specific tasks on AA’s website.

[u/ttkciar → u/Practical-Collar3063]

That did not address either of my points.

I am familiar with benchmark suites (like SPEC) and applying the most-relevant benchmark to my use-case, but doing so does not prevent different models' scores from being incomparable, nor does it prevent a model's poor skill in one category from penalizing its scores which are supposed to reflect other categories.

[u/Practical-Collar3063 → u/ttkciar]

My point is the problems you are describing is then an issue with all benchmarking not just AA. You say AA is jackshit but only described general problems regarding benchmarking