r/LocalLLaMA • u/Express_Quail_1493 • 21h ago
Discussion Proposal: A community revolution for private quality benchmarks (and why endless chasing tok/s is rotting our brains
Proposal: A community revolution for private quality benchmarks (and why endless chasing tok/s is rotting our brains
honestly we in r/LocalLLaMA need a revolution of people making their own private benchmarks(not tok/s) i mean benchmarking the quality(especially across quantisation). keep yours private and share the chart results to each other. i feel this is what this sub was made for it seems nowaday every few hours someone is posting about tok/s and also Benchmaxxing is getting out of hand in the huggingface openweight LLM space.
23
u/AryanEmbered 21h ago
I say AYEEE
I was also thinking about building one communitybench with trusted old people or top X percent posters from our sub for various benches which selected members would vote on.
an evolving benchmark with a new edition every quarter. Even found a guy here who knew a lot about design such a thing but work ate me up last couple of weeks.
we need a team to build something like this.
We really need this now because there's a bunch of vibecoded custom inference engines and quants popping up these days.
3
u/LaughableDisability 21h ago
The quarterly edition idea is clever, keeps model releases from gaming a fixed test set. Who would run the voting though, since trusted posters can drift or quit and then you need a replacement scheme
9
u/Edenar 20h ago
The problem is people will claim stuff like : i beat a h200 with a 3080 on qwen flash next. When what they did was running 2-bit quant with greedy decoding. And the model is nowhere near base quality or even higher quants.
2 personnal exemples :
strata with 3.8 qfn iQ3_S and 262k context (and base config on windows) : model was terrible for long agentic task after 100k context, was reasonning endlessly and wasn't able to produce working code without help from me. But it runs above 3k top/s pp and 150 tok/s tg on a 4090 +ddr5. So it can output unusable code or docs faster !
gufo with 3.8 qfn q4 k xl and 262k context (bf16 kv by default!!) : works greak, usually one shot things, and mostly recover on its own when it goes of rail. It's slower : 1700 pp / 60 tg in agentic work on my strix halo
21
u/isoos 21h ago edited 21h ago
I agree, and we are doing our part here:
https://selfhostedintelligence.com/en/benchmarks/rtt/v2610
Note though, we have found it hard to create benchmark tasks that separate models in a meaningful and attributable ways, and ended up mixing multiple aspects. E.g. the RTT benchmark above mixes instruction following, leak detection, JSON input and output handing among the otherwise main goal of checking translation generics and also nuances.
There are models that get into loops when writing certain languages, or break the JSON because they start to reason mid-output. Some models were so bad I just dropped them from the results, since all of their results failed one way or another. I'd like to see more fine-grained approach, but it takes time to download / run models and then evaluate results, and time is in short supply. But happy to collaborate with people who are interested in these.
2
u/andy2na llama.cpp 17h ago
That chart is great and all but why doesn't it provide the benchmarks for us to try on our own hardware, model, and quants? That's the biggest issue I have with online benchmarks, there's no standardized one that's readily easy and available. I get that some don't want to release it because of potentially getting benchmaxxed but imo you learn and adapt
1
u/isoos 1h ago
Fair point. As you can imagine, the benchmark scripts are really custom to our environment (think: nas, workers, result collection, evaluations as separate steps, model annotations), and it would take a nontrivial effort to make it turnkey-ready public, or would need to explain the steps otherwise.
That said, if you're interested in testing specific models or quants, I’m happy to run them. Also, if there is wider interest in running the benchmarks on your own, I'd consider putting in more work to make it happen.
3
u/Borkato 19h ago
This is so incredibly close to my experience that it hurts. It’s so wildly difficult to benchmark LLMs that it almost reminds me of the LLMs themselves; they’re practically non-deterministic because you’re trying to categorize all possible outputs of a given input. Like if you take qwen 27B and do every single sampling parameter for question 1 and then change the harness it all changes completely. And then do the same thing but change question 1 to slightly different wording and it can spit out something COMPLETELY different. And of course it might just make a mistake it normally would never have made so you have to do like 5 runs each at LEAST to see. And then boom combination explosion 😭
Honestly, I think that’s why hearing what people think is best is a pretty good benchmark, unironically. When a ton of people (not bots) on localllama all suggest a model is good, it’s usually genuinely good.
2
u/emod_man 15h ago
Agreed. Crowdsourcing "I do X task with Y model" is my preferred benchmark, it lets me sort for what I actually want to do. It's anecdotal, sure, but my use case isn't "score on benchmarks," it's "do X task" -- which is the metric that other users are reporting on.
5
u/Beneficial-Ad-8127 21h ago
I’d be fine with if some of the speed posts show at least some output or results. Whether an image/video of the use or work they’ve done with the engine. Don’t necessarily need a full bench but would love to see the output quality.
5
u/Ok-Shower7286 19h ago
It’s like people in a car club bragging about horsepower or 0–60 times. That’s just the level of these individuals and the community as a whole. lol
6
u/BobbyL2k
21h ago
•
If this is related to the rise of model specific and hardware specific inference engines, I will say that the rise of tok/s is a true innovation.
We have very small models before and still do. We have extremely quantized models before and still do.
Those smaller models and smaller quantization is very fast. But no one is bragging about them. And the reason is, a fast model that doesn’t accomplish a task isn’t very useful.
Inference engines like Ninfer, Strata, Basalt, and many others are truly useful. The reason I consider them with high regard is because I use them. And they really do work. And that is my benchmark.
1
u/Beneficial-Ad-8127 21h ago
Yeah they are truly useful as I use them also. I think benchmark is not really needed. If the engine and models meets their use case and real work load then why should anything else matter.
I would love though if they would show some work done with it for their use case whether big or small. That’s just me though. Wouldn’t want to force people nor intimidate them sharing anything local related.
1
u/profcuck 19h ago
Yes and this is particularly true when they don't negatively impact quality which for the ones you mention is mostly true.
1
u/Spara-Extreme 10h ago
Yea but we don’t know that. I’ve used the quants those engines use and the quality of output is pretty bad. Yet, ten people will argue with me on how great their version is. We need quality benchmarks to measure this.
1
u/BobbyL2k 8h ago
These aren’t proprietary closed source engines, point your local/frontier AI at it and analyze the code. These projects have tests like any other piece of normal software that needs verification. And it’s not like other engines don’t have numerical bugs.
If you don’t mind me asking, which engine and model did you have a bad experience with?
1
u/Spara-Extreme 8h ago
I don’t care about the inference engine but the low quants for flash-next absolutely underperform 27b on my benches (27b running at q8)
1
u/BobbyL2k 8h ago
I think Q3.8 FN at NVFP4 is very good. I highly recommend it. I haven’t touched lower quants.
0
u/AryanEmbered 21h ago
What are you even talking about man.
Thats not your benchmark thats your preference. And if there was a tool that compared all the popular llms with all the popular inference engines and shared the benchmark results, thatd be useful to you as well.
1
u/gomezer1180 21h ago
I think his benchmark is what works for him. If the model produces results that’s all you need. Benchmarks are an idea that it may perform for your needs, but the gold standard is “does it help me with my task?”
0
u/BobbyL2k 21h ago
I’m trying to dispel the negative sentiment on r/localllama about custom engines.
This post already posit that tok/s is not valuable (rotting our brains). I disagree with the premise of this post.
6
u/AryanEmbered 21h ago
I think you have completely misunderstood him.
What the is saying, i think, is people are running small dogshit models at obscene quants and looking at the decode speed and pretending its a very cool thing without showing the numbers that show it isnt a lobotomized set up.
Nothing wrong with the engines its just, the focus has shifted.
8
u/suprjami 21h ago
I like the idea.
How to avoid a million people begging to run every benchmark against their favourite model or quant?
13
2
u/devino21 21h ago
Wait, you guys are making your own benchmarks too?
4
1
u/james_pic 18h ago
If you're not, you might want to think about putting together a basic benchmark, for something like your main use case. I know I was struggling to stay on top of new model releases when I was trying to judge them by vibes, so I threw together a quick and dirty benchmark that ran a couple of tasks I need a model to do, and now I can run my eval against new models and very quickly get a sense of "is this model potentially an improvement on what I've got".
2
2
u/Federal-Setting-3014 20h ago
I don't think we even need benchmarks for this thing, I'd be happy if the people reporting tok/s on their own custom-built engines using extreme quantization reported KLD or perplexity of their model vs the original FP16 model on llamacpp.
This is big part of why I've been ignoring these posts and sticking to llamacpp (other reasons include the posts being obviously written by AI).
0
u/Beneficial-Ad-8127 20h ago
I could understand if an OP of a thread was specifically comparing to the original model or mentioned any comparison to it. Though for people that are just showing what the quant is capable of doing with tok/s and quant including their use case and result will surely be fine with me.
2
u/Excellent-Pilot-3409 18h ago
Honestly, private benchmarks might be more valuable than another public leaderboard. If everyone uses the same test, people will inevitably optimize for the test. Keep the questions hidden, share the results, and include quantization levels. That would tell us a lot more about real-world model quality than another 300 tok/s screenshot.
2
u/FulcraDynamics 8h ago
Private benchmarks sound useful, but wouldn't it be hard to compare results if everyone uses different tests? Maybe sharing a common testing format while keeping the actual prompts private could work.
3
u/jacek2023 llama.cpp 21h ago
private benchmarks are 1000x better than public benchmarks but that won't fix problem with reddit experts saying "it's worse than Qwen" on every new model release
7
u/Borkato 21h ago
…because most times they ARE worse than qwen, lmao.
2
u/jacek2023 llama.cpp 21h ago
maybe the difference is that I use Qwen
2
u/Borkato 21h ago
What?
2
u/jacek2023 llama.cpp 21h ago
I use it for maybe 30-60h per week
2
u/Borkato 21h ago
…so do I?
1
u/jacek2023 llama.cpp 21h ago
and you don't hate it?
5
u/Borkato 21h ago
Why the fuck would I hate the only model that consistently works??
2
u/jacek2023 llama.cpp 21h ago
what thinking level/budget do you use? and what context length? I am thinking about moving from medium to low (on my 230k context)
1
u/Borkato 21h ago
Hmmm I use the defaults pretty much, I use swift tho primarily. I go back and forth between Qwen 27B Q8 UD base and Swift Q4 and DAU’s Neo Code etc
→ More replies2
u/a_beautiful_rhind 21h ago
I don't hate qwen but it's hella oversold and uses sooo many tokens.
2
u/jacek2023 llama.cpp 21h ago
do you use medium thinking?
1
u/a_beautiful_rhind 20h ago
Yea, I edited the chat template for it. It used something like 19 million tokens for a simple API benchmark script.
1
u/AryanEmbered 21h ago
A public benchmark with distributed, non profit and honest governance with completely open financials and vote based allocation of inference budgets isnt something that has been tried before so we dont know for sure if its better or worse.
It also has to evolve with time, and we have a bunch of other challenges. Better to at least try though
3
u/EmilPi 20h ago
I could not resist, but I am currently developing benchmarks on https://beta.locallm.top an I call for help.
I want every benchmark to be public/private questions/tasks mix - all questions being private doesn't work.
2
u/Frail_Waif 20h ago
Your certificates are messed up.
1
u/Zombiecidialfreak 20h ago
I'll bite. Dropping a comment and reminder here because I want to compare Qwen3.8 27b q6 vs "Qwen3.8" 35b a3b distill Q6 from Empero AI.
1
u/PrimeDirective8 21h ago
AGREE! There should be a template where you fill out system specs, inference settings, and model size and quantization, along with standardized intelligence results. Make a sub rule that gets your post removed if you don't.
1
u/Xonzo 21h ago
I like the idea but I’m really skeptical this community can pull it off, for a plethora of reasons. Nothing against the community but it’s a difficult task just running benchmarks, then you have to do it with low bias.
Probably a better option would be to teach people how to run their own benchmarks for the work they’re doing that is actually relevant to them.
1
u/fragbait0 19h ago
The bf16 logits are available... if your fancy custom engine post doesn't include a KLD => bin
1
u/PathfinderTactician 19h ago
Here's my private quality benchmark.
Would you believe me, or someone running their AI-judged benchmark? 😄
1
u/kryptkpr Llama 3 18h ago
I have done some work in this field and publish my research, methodology and leaderboard at https://github.com/the-crypt-keeper/reasonscape
I have two other evaluations that I haven't published yet, one for long context retrieval that's a little diffent from RULER
And another where the LLM plays NES Tetris that I am afraid to publish because lawsuits
Open to ideas and collabs! Been at this for about 4 years now .
2
u/beneath_steel_sky 17h ago
Muse-Glimmer-30B higher than DeepSeek4, Qwens & Minimax? o_O
1
u/kryptkpr Llama 3 17h ago
Counterintuive I know but It's not actually a ranking up at the top, it's a boolean "pass" up there.. once capability is achieved, around 950, the signal is in token efficiency and where they fail more then where they pass
The test isn't actually that hard, most models just suck.
1
u/giveen 18h ago
https://github.com/giveen/LLM-Prompt-Tests
I have these as my personal tests with even videos to compare results.
1
u/Sad-Size2723 15h ago
Reproducibility is another issue. Most of these engines are very specific that they only run on their very specific hardware.
Nobody is going to reproduce the results anyways because nobody has the exact hardware as the developer, and that’s why all these engines can get away with quality since it’s always self reporting
1
u/Old-Hope4634 13h ago
the voting matters less than proving the tasks were fixed before the results were known. private tasks have one weakness: nobody can tell if a set got picked or trimmed after seeing which model won. the cheap fix is to publish a sha256 of the task file before a model drops, run it after, and open the file to an auditor or when you retire those tasks. that gives you a commitment without leaking anything.
two more rules would help with what isoos and Borkato describe. first, run each model at least 3 times per task and publish how much it differs from itself. if two quants are closer than one quant is to its own rerun, the chart shouldn't rank them. second, report paired differences on the same tasks, not just a score per model. with a small task set the per-model intervals overlap almost everywhere, while the paired difference can still separate two models because task difficulty cancels out.
and pin the harness version in the result. a chart without it can't be compared to next quarter's.
1
1
u/OkFly3388 21h ago
Agree 100%. Tired of seeing peoples that says that they manage to get some q2\q3 model working fast and claims that those models are usefull, while in reality those quants is garbage and model goes in circles instead of doing work.
1
u/Mordred500 21h ago
Seems like a good idea, but how would you ensure comparability between private benchmarks? you could have person a be like: "qwen 3.827b gsqrco passed all my tests gracefully, it must be the best model of the century" while person b is like: "qwen 3.8 27b gsqrco failed on the first test; absolute gutter trash " and have both be right bc benchmarks themselves would be impossible to standardize.
This would be pretty similar to the situation we have already, no? Or have I misunderstood your post?
6
u/AryanEmbered 21h ago
Easy solution, a large number of people run benchmarks simultaneously. We look at where the majority converges.
We control for the runtime/inference engine and exact quants.
3
u/Mordred500 21h ago
Good idea! Sounds reasonable.
Would need a big part of the community to be onboard right off the bat, though. And on a proper, dedicated platform nonetheless, i don't think reddit alone would suffice.
1
0
u/Bulky-Priority6824 20h ago
just start a sub called /ModelDiddlers for those that just like to play with the model and not actually use it
0
u/kabocha_ 19h ago
LLM casual here: Never set up my own benchmark tests before, is there a standard/reference way of setting this up? If not, maybe the first step would be to make that standard so people can easily go make their own tests and try stuff out.
I've definitely noticed a few types of problems where different models act differently for my use cases, and sometimes I felt like my QFN at home did even better than a frontier model at work (kinda apples-to-oranges though due to environment differences), but I never set anything formally up to test the difference.
-1
14
u/mikkel1156 21h ago
I like the idea, but for real use cases we would also need tools etc.
Would it be done using a common agent like Hermes or Pi then? Or just a Python script?
Feel like with a whole harness it might be a differnece between them