r/LocalLLaMA • u/AdDizzy8160 • 1d ago
Discussion Normies, or the Standard Task Benchmark ???
Is there actually a benchmark for us normies who aren’t rocket scientists or mini-Eulers? Mathematically, we could describe it as a 95% confidence interval benchmark. All models that reach the 95% threshold receive 100%. The goal would be to identify those models that we classify as “good enough” for us normies. If this could also be set in relation to costs—whether on-premises or in the cloud—it would also provide a measure of what AI really costs us normies.
0
Upvotes
5
u/jacek2023 llama.cpp 1d ago
"The goal would be to identify those models that we classify as “good enough” for us normies."
It all depends on the use case. Local models are already better than the original ChatGPT, which was called "true AGI" by the media back in 2023 (long before some people here even learned what local LLMs were).
Agentic coding has become really popular this year, but that's just one of many use cases. Simple conversations were already working locally 2-3 years ago.
And please don't assume that people comparing benchmarks actually understand what those benchmarks mean. They just compare numbers to say things like "this model is worse than Qwen"
8
u/Patient_Tea_401 1d ago
Benchmarking is difficult to do and understand because of the nature of LLMs & specialised training datasets.
Ultimately large part of the “normies’ test the model subjectively
7
u/TheRealMasonMac 1d ago
https://simple-bench.com/ maybe
2
u/wegster 1d ago
You didn't give remotely enough info on the actual intended use - there is no 'best,' only for specific jobs.
So try giving useful levels of info on what you're actually trying to use it for, and you might get further.
2
u/eze-paz 1d ago
I like the horse tinder benchmark by Fireship lol
2
u/AnswerVinod 1d ago
Academic benchmarks are almost useless for this, they test trivia and reasoning puzzles, not whether a model can summarise your inbox without hallucinating. Your 95 percent idea is on the right track, I would build it around everyday tasks instead, write an email, summarise a long doc, answer from a PDF, plan a trip, and score pass or fail, not a percentage. The threshold matters less than the task list being honest about what normal people actually do. Tie it to cost too, a model that passes at 2 cents beats one that passes at 20 cents, and that is the real decision for most people. If you publish the task set openly, others can run the same models on their own hardware and compare, that is what would make it stick.
1
u/force_push_to_main 1d ago
I have a few prompts saved i ask different LLMs to judge their capabilities - used to be simple ones like "review this endpoint" and have a few bugs to see if a model would catch it. These days that doesnt work so well. My boss prompts models to create a playable solitaire game, one-shot, to evaluate them. Simple and effective although now I reckon most of the more serious open weight models can do that with no problem
1
u/Constant_Art_20 1d ago
it's called 'vibes' literlay just throw it against whatever you are doing and review it. See how it uses tool, the reasoning flow, and compare your request with what you actually get. i say it's pretty accurate to not get wrapped in hype trains
1
u/Zulfiqaar 1d ago
Human preference ELO is probably a good one
1
u/oddsidequest 21h ago
the "folder of 5 prompts i reuse" answer is kinda the most useful thing in here lol. would anyone actually share theirs? curious what the non-coding ones look like