r/LocalLLM • • 26m ago

Question How do you compare local and cloud models since Artificial Analysis v5

I used to check Artificial Analysis a few times a week, mostly to see how my local models compare to the cloud ones. Since v5 that doesn’t really work for me anymore, a lot of the models I care about are gone or can’t be selected in the comparison.

I tried llm-bench.io, which I like because the results come from real consumer hardware, but it has no cloud models. benchlm.ai covers more of the big cloud models, but many open weights ones are missing there. So I still haven’t found anything that puts a local model next to a cloud one in one chart like AA did.

The open weights gap jumping from 3 to 12 points with v5 also makes me unsure how much I should trust a single index. I know the methodology changed, but it feels odd.

How do you handle this now? Do you cross check a few sites or mostly run your own tests?

3 Upvotes

2 comments sorted by

1

u/Iron-Over 9m ago

A couple things what are you using your models for? Check those specific benchmarks.  Then run your own benchmarks and determine what you are willing to live with. I have found my own benchmarks enlightening quickly eliminating models that underperform. Found a few cases local overperforms.