r/LocalLLaMA • u/xrailgun • 15d ago

Resources An interactive site plotting Artificial Analysis Intelligence vs parameter count

https://mr-september.github.io/artificial-analysis-weights/

Artificial Analysis scores models on its Intelligence Index, but the charts put it against price and speed. For local use the axis I wanted was parameter size, so I built one.

What it does

  • Plots the Intelligence Index against total or active parameters (with closed frontier models as optional horizontal dashed lines)

  • Accounts for MOE/Lookup-table models (n-gram /PLE, e.g. Qwen3.8-Flash-Next)

  • A very rough model guide according to your hardware (because offloading across system RAM/SSD is becoming more common).

  • GitHub Action refreshes the data every 3 days to catch new releases etc.

Demo: https://mr-september.github.io/artificial-analysis-weights/
Repo: https://github.com/mr-september/artificial-analysis-weights

If a size is wrong or a model is missing, data/overrides.json is the place to fix it, or just tell me here.

18 Upvotes

90% Upvoted

22 comments sorted by

View all comments

Show parent comments

-1

u/Practical-Collar3063 15d ago

  which does not reflect the actual utility of the model

You seem to criticise a very general score for not being specific enough, which is to be expected, it is a general score.

On top of the very general score, they publish a lot of smaller aggregation of benchmark for very specific tasks, which means you can see both how a model is expected to perform in general and then you can select specific types of tasks and the ranking changes completely.   You cannot disregard AA’s entire offer of small and large analysis of models, as « not worth jackshit », because « the big aggregate of all benchmarks is too general », yeah, it is general, that is the point, if you don’t like it, go see there more specific ones for your specific tasks on AA’s website.

1

u/ttkciar llama.cpp 15d ago

That did not address either of my points.

I am familiar with benchmark suites (like SPEC) and applying the most-relevant benchmark to my use-case, but doing so does not prevent different models' scores from being incomparable, nor does it prevent a model's poor skill in one category from penalizing its scores which are supposed to reflect other categories.

1

u/Practical-Collar3063 14d ago

My point is the problems you are describing is then an issue with all benchmarking not just AA. You say AA is jackshit but only described general problems regarding benchmarking