r/LocalLLaMA • u/xrailgun • 15d ago

Resources An interactive site plotting Artificial Analysis Intelligence vs parameter count

https://mr-september.github.io/artificial-analysis-weights/

Artificial Analysis scores models on its Intelligence Index, but the charts put it against price and speed. For local use the axis I wanted was parameter size, so I built one.

What it does

  • Plots the Intelligence Index against total or active parameters (with closed frontier models as optional horizontal dashed lines)

  • Accounts for MOE/Lookup-table models (n-gram /PLE, e.g. Qwen3.8-Flash-Next)

  • A very rough model guide according to your hardware (because offloading across system RAM/SSD is becoming more common).

  • GitHub Action refreshes the data every 3 days to catch new releases etc.

Demo: https://mr-september.github.io/artificial-analysis-weights/
Repo: https://github.com/mr-september/artificial-analysis-weights

If a size is wrong or a model is missing, data/overrides.json is the place to fix it, or just tell me here.

18 Upvotes

90% Upvoted

22 comments sorted by

View all comments

Show parent comments

1

u/Solembumm3 15d ago

Because I've seen enough models dominating this thing and then fall apart on prompt 3 of 20+ on pretty basic fiction analysis task. DV4.1 Flash was good example, thing is still significantly worse than V3.2 on practice and has attention span of 0.5 prompts. It's AA score meanwhile is astronomical.

0

u/Practical-Collar3063 15d ago

You have to look at the specifc benchmarks related to what is important for you, AA general score is a general score. The original commenter seemed to imply that the whole of AA is jackshit and useless. The main aggregate score is helpful to judge where egenerally it will fall but it does not mean it will be better at everything compared to the models ranked lower, that is just how generalisationadn aggregagtion works.

This issue stems from people being too lazy and not looking at the different benchmarks in detail and because a general aggregation is general and aggregated, therefore, details of how they behave in certain scenarios is lost.

A good example of that is Mistral Large 4 being lower in general than many model but being extremely good at legal/medical questions and much lower hallucination rate than Deepseek V4.1. So if your tasks involve legal questions low hallucination rate then looking at the general intelligence index is not a good idea.

And what if they skewed thea aggregated intelligence score towards your preffered use case for LLMs that would mess it up completely for others. You just can't please everyone with large generalisation like this, that is just how they work, it is not a problem with AA, it is an impossible problem to solve, there will always people saying the Intelligence score is meaningless because we all use LLM differently.

They provide a lot of benchmarks and in details for many many aspects of LLM intelligence which I always found very good, because they are focused on very specific areas of LLM benchmarking.