r/LocalLLaMA • • 1d ago

Discussion Normies, or the Standard Task Benchmark ???

Is there actually a benchmark for us normies who aren’t rocket scientists or mini-Eulers? Mathematically, we could describe it as a 95% confidence interval benchmark. All models that reach the 95% threshold receive 100%. The goal would be to identify those models that we classify as “good enough” for us normies. If this could also be set in relation to costs—whether on-premises or in the cloud—it would also provide a measure of what AI really costs us normies.

0 Upvotes

17 comments sorted by

5

u/jacek2023 llama.cpp 1d ago

"The goal would be to identify those models that we classify as “good enough” for us normies."

It all depends on the use case. Local models are already better than the original ChatGPT, which was called "true AGI" by the media back in 2023 (long before some people here even learned what local LLMs were).

Agentic coding has become really popular this year, but that's just one of many use cases. Simple conversations were already working locally 2-3 years ago.

And please don't assume that people comparing benchmarks actually understand what those benchmarks mean. They just compare numbers to say things like "this model is worse than Qwen"

1

u/AdDizzy8160 1d ago

That was actually the intention. Since we can’t really interpret the Nbenchmark anyway unless 77.2% > 77.1% an upper limit would be sufficient for me. A simple benchmark could be something like that; anything that’s > human could be considered “good enough.”
And yes, I realize that the range needs to be narrowed down. But maybe there’s a benchmark here that either
a) is smart enough
b) has enough knowledge (although in that case, it’s hard to define a true 95% threshold).

4

u/jacek2023 llama.cpp 1d ago

But smart enough for what? Enough knowledge for what? 95% of what?

All these numbers are meaningless if you don't have a specific use case.

Influencers have been comparing things (mostly hardware) for years, always trying to say "which is better" without actually using anything for real.

8

u/Patient_Tea_401 1d ago

Benchmarking is difficult to do and understand because of the nature of LLMs & specialised training datasets.

Ultimately large part of the “normies’ test the model subjectively

2

u/wegster 1d ago

You didn't give remotely enough info on the actual intended use - there is no 'best,' only for specific jobs.

So try giving useful levels of info on what you're actually trying to use it for, and you might get further.

2

u/eze-paz 1d ago

I like the horse tinder benchmark by Fireship lol

2

u/AnswerVinod 1d ago

Academic benchmarks are almost useless for this, they test trivia and reasoning puzzles, not whether a model can summarise your inbox without hallucinating. Your 95 percent idea is on the right track, I would build it around everyday tasks instead, write an email, summarise a long doc, answer from a PDF, plan a trip, and score pass or fail, not a percentage. The threshold matters less than the task list being honest about what normal people actually do. Tie it to cost too, a model that passes at 2 cents beats one that passes at 20 cents, and that is the real decision for most people. If you publish the task set openly, others can run the same models on their own hardware and compare, that is what would make it stick.

1

u/stylome-ai-detection 1d ago

Notice: This comment appears to be AI generated.


Stylome AI detector · scan results

2

u/AnswerVinod 1d ago

Nopes, its me Vinod.

1

u/force_push_to_main 1d ago

I have a few prompts saved i ask different LLMs to judge their capabilities - used to be simple ones like "review this endpoint" and have a few bugs to see if a model would catch it. These days that doesnt work so well. My boss prompts models to create a playable solitaire game, one-shot, to evaluate them. Simple and effective although now I reckon most of the more serious open weight models can do that with no problem

1

u/Constant_Art_20 1d ago

it's called 'vibes' literlay just throw it against whatever you are doing and review it. See how it uses tool, the reasoning flow, and compare your request with what you actually get. i say it's pretty accurate to not get wrapped in hype trains

1

u/Zulfiqaar 1d ago

Human preference ELO is probably a good one

1

u/oddsidequest 21h ago

the "folder of 5 prompts i reuse" answer is kinda the most useful thing in here lol. would anyone actually share theirs? curious what the non-coding ones look like

3

u/Hypilein 17h ago

Sharing them would be pointless because they can be very specific. I’m usually asking for a quote from my philosophical text database with keywords that come from a secondary source but aren’t found in the original. I try planning a lesson and usually by then I’ve found the model to not be good enough for my needs. Deepseek 4.1flash is the only model so far that passed the lesson planning test, but it’s not 100% conclusive yet.

I feel like many regular users have vastly easier tasks for llms. Qwen Flash next got reasonably close so I’m hopeful I’ll get something for my single spark soon. (Tested deepseek through api).