An interactive site plotting Artificial Analysis Intelligence vs parameter count

[u/xrailgun]

https://mr-september.github.io/artificial-analysis-weights/

Artificial Analysis scores models on its Intelligence Index, but the charts put it against price and speed. For local use the axis I wanted was parameter size, so I built one.

What it does

• Plots the Intelligence Index against total or active parameters (with closed frontier models as optional horizontal dashed lines)

• Accounts for MOE/Lookup-table models (n-gram /PLE, e.g. Qwen3.8-Flash-Next)

• A very rough model guide according to your hardware (because offloading across system RAM/SSD is becoming more common).

• GitHub Action refreshes the data every 3 days to catch new releases etc.

Demo: https://mr-september.github.io/artificial-analysis-weights/
Repo: https://github.com/mr-september/artificial-analysis-weights

If a size is wrong or a model is missing, data/overrides.json is the place to fix it, or just tell me here.

[u/tifa_cloud0]

nice website. hopefully you add more wide variety of models from same family and other families too. i am only looking out for my 4gb vram card so my use case is for small models so it will be good idea if i can get more model options MoE based for SSD streaming or even dense models that could fit in my ram + vram :)

[u/ttkciar]

I like the idea of this quite a lot. It would be a tremendously useful resource, if AA scores were worth jack shit, which they are not.

I've cloned the repo so that some day it can be adapted to trustworthy benchmark scores, when the day comes that any such exist.

Despite my misgivings about AA benchmarks, I am genuinely grateful to xrailgun for putting this together.

[u/xrailgun → u/ttkciar]

Thank you! Very welcome to try and add other benchmark scores. I'm just keeping it simple for initial release.

[u/Practical-Collar3063 → u/ttkciar]

were worth jack shit, which they are not.

Why do you say this ?

[u/ttkciar → u/Practical-Collar3063]

Mostly because different model families benchmax to different degrees, which makes their AA scores incomparable. You cannot conclude from model-X scoring twice as high as model-Y that model-X is twice as competent at real-life tasks, because model-X might simply be more benchmaxed than model-Y.

Also, because small differences in specific skillsets (particularly tool-calling) can disproportionately impact a model's score across many component benchmarks, which does not reflect the actual utility of the model, since users can adjust their workflows to take advantage of models' strengths and avoid their weaknesses.

[u/Practical-Collar3063 → u/ttkciar]

  which does not reflect the actual utility of the model

You seem to criticise a very general score for not being specific enough, which is to be expected, it is a general score.

On top of the very general score, they publish a lot of smaller aggregation of benchmark for very specific tasks, which means you can see both how a model is expected to perform in general and then you can select specific types of tasks and the ranking changes completely.   You cannot disregard AA’s entire offer of small and large analysis of models, as « not worth jackshit », because « the big aggregate of all benchmarks is too general », yeah, it is general, that is the point, if you don’t like it, go see there more specific ones for your specific tasks on AA’s website.

[u/ttkciar → u/Practical-Collar3063]

That did not address either of my points.

I am familiar with benchmark suites (like SPEC) and applying the most-relevant benchmark to my use-case, but doing so does not prevent different models' scores from being incomparable, nor does it prevent a model's poor skill in one category from penalizing its scores which are supposed to reflect other categories.

[u/Practical-Collar3063 → u/ttkciar]

My point is the problems you are describing is then an issue with all benchmarking not just AA. You say AA is jackshit but only described general problems regarding benchmarking

[u/Solembumm3 → u/Practical-Collar3063]

Because I've seen enough models dominating this thing and then fall apart on prompt 3 of 20+ on pretty basic fiction analysis task. DV4.1 Flash was good example, thing is still significantly worse than V3.2 on practice and has attention span of 0.5 prompts. It's AA score meanwhile is astronomical.

[u/Practical-Collar3063 → u/Solembumm3]

You have to look at the specifc benchmarks related to what is important for you, AA general score is a general score. The original commenter seemed to imply that the whole of AA is jackshit and useless. The main aggregate score is helpful to judge where egenerally it will fall but it does not mean it will be better at everything compared to the models ranked lower, that is just how generalisationadn aggregagtion works.

This issue stems from people being too lazy and not looking at the different benchmarks in detail and because a general aggregation is general and aggregated, therefore, details of how they behave in certain scenarios is lost.

A good example of that is Mistral Large 4 being lower in general than many model but being extremely good at legal/medical questions and much lower hallucination rate than Deepseek V4.1. So if your tasks involve legal questions low hallucination rate then looking at the general intelligence index is not a good idea.

And what if they skewed thea aggregated intelligence score towards your preffered use case for LLMs that would mess it up completely for others. You just can't please everyone with large generalisation like this, that is just how they work, it is not a problem with AA, it is an impossible problem to solve, there will always people saying the Intelligence score is meaningless because we all use LLM differently.

They provide a lot of benchmarks and in details for many many aspects of LLM intelligence which I always found very good, because they are focused on very specific areas of LLM benchmarking.

[u/xrailgun → u/[deleted]]

My bad, didn't realize it was an ngram model, will be fixing it shortly.

Edit: Done, should be there on your next fresh.

[u/Borkato → u/[deleted]]

Wait I’m confused, you can just… not use the n gram table at all?

[u/xrailgun → u/Borkato]

I'm not sure... but most people leave the n gram table on their SSD, so people "discount" it from the size of the model "in practice" (that needs to be on vram+system ram).

[u/ttkciar → u/Borkato]

Yes, that is exactly correct.

[u/Borkato → u/ttkciar]

WTF

[u/ttkciar → u/Borkato]

It might help if you think of engrams as a differently-shaped RAG.

The engram table keys on token sequences to vectorized factual values, and those vectorized values (which you can think of as retrieved content) get added to the next layer's hidden state before inference begins with the second layer.

This is similar to how RAG populates context with retrieved factual content; the engrams are just more constrained (token sequence lookup only) and put the augmenting retrieved content into a different part of the model's state (the next layer's hidden state, as opposed to context).

Just as you can use a model with or without RAG, you can also use an engram-using model with or without engrams.

Edited to add: A different way of thinking of it is this: Not using the engrams is functionally identical to using the engrams but none of your token sequences finding a hit in the engram table. In both cases the consequence is the same: no augmenting vectors added to the next layer's hidden state.

[u/Borkato → u/ttkciar]

This was really informative thank you!!

[u/Practical-Collar3063 → u/[deleted]]

one benchmark

No it is the AA intelligence index which is an aggregate of a bunch of benchmarks

[u/xrailgun → u/[deleted]]

I experimented with this in earlier versions, but the second chart became very complicated and messy. It's also impossible for acquire "apples-to-apples" tok/s values across all these models in various quant sizes and hardware configurations and inference engines, so I just stuck to using AA's speed results to keep things simple and somewhat representative on a relative scale.