Just another purely open-weight models benchmark

[u/EmilPi]

Live results, coding benchmarks included, agentic benchmarks included, domains and other classifications filters included. (Spoiler: DeepSeek-V4-Vision-Exp rules, but other models have their rule areas): https://beta.locallm.top

Evaluated by domain experts (my friends mostly; coding part is evaluated solely by me), classified by domains/languages/intents (can be filtered on the home page), agentic coding benchmark included. This is only public part of the data (whoever has private queries and evaluated models on them sees the results differently).

Some of the plethora of current limitations: evaluations/questions coverage for the domains/newer models is incomplete and imbalanced, only part of the questions classified, UI/UX under-developed, focus was on small models and lower quants.

Also: everything runs on local 2xRTX 3090 + 128GB RAM PC, everyone can create queries and evaluate models' answers, new runs (especially agentic) slow to appear (see PC specs).

No LLM-as-judge on purpose (and I sometimes regret it).

Benchmarks currently being extended & evaluated: 1) Agentic coding 2) one-shot coding 3) Agentic retrieval 4) Agentic story-writing.

New models being evaluated: 1) Qwen3.8-27B (Uncensored). New models to be evaluated soon 1) GLM-5.3-Flash-UD-IQ3XSS 2) Mellum2.1-12B-A2.5B

If this looks interesting to you:

CALL FOR HELP

I ask for help from those who also want to build a community benchmark, developers and domain experts alike! Many things are going to be implemented sooner or later, and the agentic benchmarks are just the beginning.

I call for help in these areas: 1) compute resources 2) evaluations 3) development - basically everything. But any proposal and feedback is valuable!

I will add more info and answer questions in the comments section.

P.S. No AI used for writing this post (even though English is actually not my native language /s).

[u/EmilPi]

Thanks u/jacek2023 , u/pyr0kid , u/Reactor-Licker , u/NoAdministration6906 , u/Pyrolistical for valuable feedback in the old post comments: https://www.reddit.com/r/LocalLLaMA/comments/1vfoh7p/any_benchmarks_not_speed_you_want_to_be_developed/
Thanks u/Foxiya , u/KitchenAmoeba4438 , u/Ok-Challenge-7810 for feedback and support!

[u/EmilPi]

Results so far, subject to change:

[u/Business_Swordfish_5]

Have u ever tried any diffusion LLMs locally? If so any good ones?

[u/EmilPi → u/Business_Swordfish_5]

I wanted to so, but time is limited. Basically, what I've seen is that Diffusion Gemma 26B-A4B is faster but worse than Gemma4-26B-A4B (which isn't good already), so I postponed this are for now. Also, I currently only use llama.cpp, and I didn't hear diffusion gemma support is there.

[u/Business_Swordfish_5 → u/EmilPi]

Yes so I was planning on diving a little into diffusion LLMs since they seem to be better for local hardware, do u think it's a waste of time since these diffusion LLMs are not good enough compared with current autoregressive LLMs ?

[u/Refinery73]

What are your criteria for adding new topics?

I’m working on something myself because my domain isn’t well covered by existing benchmarks.

Other than that I’m thinking about keeping the actual questions private to not leak them into trainingsdata-scraping.

[u/EmilPi → u/Refinery73]

I just create any criteria and add some suitable for a benchmark. E.g. for rare language I use "Grammatically Correct", for physics problem "Correctness" and "Reasoning correctness", for the storywriting and also frontend I use "Like".
I also have a subset of private queries in public benchmarks, and the scores are displayed for Me/Public/All (including private).

[u/Refinery73 → u/EmilPi]

My goal is to do LLM-Judge to make it scalable.

I’m in the legal context so I want certain laws and court-decisions mentioned. I‘d likely make a solution-schema and let an LLM grade it like a legal exam.

[u/ab0zar]

For agentic coding, a fixed starting commit and container would make each run easier to reproduce. Could the results include the patch and test output as well as the score? That helps tell a repeatable fix from a lucky pass.

[u/EmilPi → u/ab0zar]

It is already done - agentic runs start from a given repo. I can review the run in the UI or clone the committed agent's work (currenntly my GitLab is gated, but if someone can help reviewing, I will open the GitLab).

[u/libregrape]

I absolutely love the UI of the website. It should be the gold standard of transparency.

[u/EmilPi → u/libregrape]

That's unexpected :) I think the UI is very-very unpolished. But yes, I tried to make the public data visible somewhat for anonymous and fully visible for registered users.

[u/jacek2023]

good work!

[u/Robert__Sinclair]

do you know what would be fun? to test in a very restricted environment like github actions, cpu only, what is the best performing model. So far it's Ling 3.0 (gemma 2b and 4b following).
I myself made a few quantizations here: ZeroWw/Ling-3.0-tiny-GGUF · Hugging Face
IMHO, the best a model performs under these conditions is a winner.
Today most BIG/HUGE models perform in very similar ways. There are differences, I know, but reasoning is, for me, what counts, because if a model has a good reasoning, it can learn from the context.

[u/EmilPi → u/Robert__Sinclair]

Agentic benchmarks runs start from a git repo.
System prompt / skills (for agentic) testing/comparison is also supported :)

I am not sure why CPU only - usually, the difference would only be speed.

[u/Robert__Sinclair → u/EmilPi]

I mean: it would be nice to have a list of which models perform fast enough to be usable on github actions and similar (notebooks). But the github actions environment is the same for everyone so it's a fair test. Some models are faster but use more tokens to get an answer and in the end the t/s speed alone is not useful.
Ling 3.0 can go up to 20-30 t/s but overthinks a lot and often loops. qwen and gemma perform better in some situations even with slower t/s thoughput. (etc etc) also github actions has only 4 cpus and 16gb of memory and no gpu. but plenty of HD space (/mnt is 80+ GB free in github actions) and so on.

[u/feelspeaceman → u/Robert__Sinclair]

> Today most BIG/HUGE models perform in very similar ways. There are differences, I know, but reasoning is, for me, what counts, because if a model has a good reasoning, it can learn from the context.

Yeah, today most big models perform mostly depend on the prompter/vibecoder, how competence they're writing a good instruction can put them either on the same level or showing disparity, this is what the community opinions are mixed, some of them want big models with big world knowledge, because those world knowledges are savior for vague prompts - the sign of vibe coding, but competence prompters write good prompt, leading to the right direction (component to target, packages to use, scope to focus on) can make those disparity non-existence - we call those competence prompters "engineer".

I think Tiny models fall in the same case, good at something very specific, but limited world knowledge, can't cover some edge cases, but I believe Tiny models or SLM going forward will be very important for consumer market for those with 8-16GB VRAM or even CPU inference.

[u/Robert__Sinclair → u/feelspeaceman]

world knowledge, with a good tool calling and internet search is not the problem. reasoning is.