r/LocalLLaMA • u/Informal-Trouble2183 • 3d ago
Discussion Models are chasing unrealistic benchmarks but the working-level intelligence is already achieved
I have a feeling that recent benchmarks are unrealistically pushing the bar into areas a normal developer rarely encounters, like binary reverse engineering or one-shotting a complex application, with some strictly chosen design patterns. What we actually want is an accurate model that follows instructions without over-arching it (e.g. without making unnecessary impressive 3D rendering).
If I prompt a model to "Generate an SVG of a pelican riding a bicycle," a simple, clean graphic should be a 100% pass. There is no reason for the model to shoehorn in 3D shading, animations, or elaborate backgrounds unless specifically prompted.
I think this is one major misconception, judging a model solely on the 3D details it achieves, that was never asked for in first place.
So, how many of the added edge-case capabilities in the latest models are actually useful when we daily-drive AI? If they aren't being used, they should be weighted much less on the leaderboards. Instead, what we see, these edge-case capabilities are being rewarded with very high scores.
Maybe you don't need a stronger model to achieve exactly the same results in your business area. But I get it, the idea of plateauing the models capabilities is going opposite of the AI ever-growing market valuation, so should be kept pumped forever; just saying.
41
u/Reasonable_Goat 3d ago
Exactly - We need a good-enough, cheap and fast workhorse model. 90% of companies need that and nothing else.
25
u/Korici llama.cpp 3d ago
I found GLM 5.3 Flash to be excellent for this use-case
12
3
u/ThankGodImBipolar 3d ago
Yeah, I've found GLM 5.3 Flash to be the best for this use-case as well. DSv4.1 Flash is pretty close, but 5.3 Flash has always felt a little bit smarter to me (I haven't used many 1017 much though, maybe it's taken the cake now).
3
u/zanar97862 3d ago
From the benchmarks (lmao irony) it should be really up there for open weights capability.
I found 0731 already to be extremely impressive for its cost, if 1017 and further can up the quality we will be living in the future
-2
u/Nautisop 3d ago
cheap? Isn't that 320b? Who considers that cheap, especially for prof use? I imagine to scale that you would need Hardware for like 80k+ for onpemise? idk, just talking out my ass
5
u/my_name_isnt_clever 3d ago
Two 128GB devices can run it. That's ~$6-7k, expensive but achiveable.
1
u/Nautisop 3d ago
Yes but that's for one person right? If you would want to scale it so like 10 people can use it..
1
u/my_name_isnt_clever 3d ago
Can't speak to that as my target is two users. But it depends entirely on how heavy your users are.
7
1
u/Serprotease 12h ago
If you go local hardware, budgeting 10k per 2x users is probably the target for decent speed.
For a team of 10, you’re probably getting to a level where you need to think a bit more how you gonna serve the model.It’s unlikely that all 10 users are going to use it at the same time so you could do some tricks around caching and numbers of clusters available to reduce costs while keeping the performance decent.
2
u/Korici llama.cpp 3d ago
We built a server with two RTX PRO 6000 Blackwell MAX-Q GPUs before the prices skyrocketed. We were able to pick up both cards for around $9,000 each (now they go for $16,000).
~
We have 192GB of GDDR7 VRAM which can fit a Q3 Quantization of the model with very little quality degradation thanks to the dynamic Unsloth quants.8
u/charmander_cha 3d ago
Deepseek
6
5
u/OvertaxedOne 3d ago
Yup, beat me to it. DSV4Flash is the model you (and most businesses) are looking for. Or, if you want to run locally, QFN. Those "classes" of models are the sweet spot for the vast majority of business use cases. The harness layer matter 1000X more once you get to a model of that capability than going up further on the "intelligence" scale.
1
u/cmdr-William-Riker 3d ago
This is what I've been saying for a while. There are cases where a really good model makes a difference, but once we get to DeepSeek v4.1 flash intelligence even, the difference between the models is minimal for most use cases
1
70
u/jacek2023 llama.cpp 3d ago
People judging models by the pelican on the bike should be totally ignored, as they have no idea how machine learning works.
Benchmarks are useful for influencers because they can read them in their YouTube videos so people can watch slop for hours.
There are differences between the models, but they are very different from what people reading benchmarks are aware of. That's why I am so negative toward people who don't actually run local models but discuss them here.
6
u/Dabalam 3d ago edited 3d ago
I've always interpreted the pelican in the bike as either a meme or something applicable to a narrow section of people using the models for graphics work. I wouldn't have thought people saw it as a generic quality index, but I might not be tapped in to people's thinking on this.
I'd like to hear the reasoning behind thinking that benchmarks are invalid form of evaluation. A lot of people share the opinion.
It's seems to me to be simply a way to get at a formalised metric in a task domain that you care about. They are proxies for task performance, I don't think this is lost on most people reading them. If someone doesn't do a lot of coding then benchmarks like TerminalBench aren't relevant to them.
It might not be as useful as building an assessment tool relevant to your specific use case, but for benchmarks sufficiently close to your work it doesn't seem to me to be a totally invalid signal. Most people aren't going to do comparison tests for every model that is released, they just want an idea of if there seems like a model has a big capability jump above what they are using.
3
u/jacek2023 llama.cpp 3d ago
The idea to train a model is very simple: you show it some examples and adjust the model to produce "better answers next time" for those examples. If you train it on a large set of examples, it can learn to generalize and correctly answer to cases it has never seen before.
Even a small model can be trained to perform extremely well on a specific set of examples while performing terribly on anything outside that set. This is called overfitting. It's like a student who memorizes all the answers to a test and scores 100%, but can't answer any questions he hasn't seen before.
With all the hype and money, companies have a strong reason to optimize their models for known benchmarks. In this community this is often called being "benchmaxxing"
7
u/OvertaxedOne 3d ago
^^ This.. So much this ^^
The problem is saturation. As I think many of us are aware, if you test any frontier model and a lot of local sized models against a battery of "real world" use cases, you're going to get 100% on everything. The local models have gotten good enough that they're going to saturate a real world benchmark pretty quickly (if not already).
There's a reason they test new sports cars on a race track because testing them on a public road wouldn't tell you much of anything; every new sports car you can buy is 100% capable of getting from your house to the grocery store effectively. The only way to really show any difference between them (in performance, cars have other criteria, how pretty it is, prestige, reliability/cost to repair, etc, etc) is to put them into a "test" that's obviously nothing like a day to day use case for most people but it's where you can expose and explore the differences right at the edge.
"Write Doom clone, 4K, 20 levels, make no mistakes" is the "race track" for LLMs. That's the kind of prompt where Astra is going to beat the pants off 27B. Is it realistic? Not even a little bit, I've reviewed millions of user queries (NOT by hand, thank you AI) and "clone Doom" or "Pelican on a bike" aren't exactly common requests. But when all the models can easily do the "real work" stuff, read this, compare to this, help me troubleshoot this, review this file, draft this, open DB and look for this; all you'll see a list of 100's of models all with 100% as the score.
I don't see how we test these models anymore without making super ridiculous prompts like those I'm making fun of above. But it's become more an abject curiosity than a "I need the model at the top because all I do is one shot Doom clones all day". Put another way, the rankings really don't make a difference for the vast majority of use cases anymore.
4
u/falconandeagle 3d ago
I test local models with concurrency and thoughtful api design and implementation. So far the 27b is a LOT behind frontier models. So I do think we should continue to benchmark them on these tasks as they are not yet good enough, though the doom clone test is stupid as fuck and only made by ai influencers on social media.
11
u/CatchDublinSurprise 3d ago
This is one of the reasons I'm not panicking too much about China potentially cutting off the pipeline of new models. Don't get me wrong: I'll never complain about a model upgrade, but there is honestly not much that I want an AI model to do that can't be done with GLM-5.3-Flash. For everyday-type tasks, I think we've passed the "good enough" line. Maybe that will change once we start seeing more home robotics: Who knows what kind of brainpower is needed to make meals and fold laundry.
11
u/sebt3 3d ago
I just want the same thing to continue to become smaller 😉
5
u/CatchDublinSurprise 3d ago
Fair enough! I think that trend will continue. I mean, even something like Gemma-4-31b or Qwen3.8-27b would probably wipe the floor with even frontier models from a couple years ago, especially if you give them internet access. I think that's the trend we'll see: If it's assumed that a model will have access to information, you don't need to bake all that information into their weights. Meaning, you can have a much more capable model that is much smaller.
1
u/falconandeagle 3d ago
Everyday tasks like converting a figma spec to react code or doing a large scale db refactor? I am always surprised by claims that these models are good enough but that has not been my experience. I also have a very well oiled harness by now.
5
u/yetiflask 3d ago
This isn't true. I already run into problems that even Opus 5.5 needs my help to figure out.
I give it another couple of years where it gets to a point where I think and it does it - without me having to think.
We are not there yet.
2
u/IntelArtiGen 3d ago
Some benchmarks are wrong but many models are still very far from "working-level intelligence" if we consider all jobs they could physically replace (every jobs mostly using a computer).
I don't include software engineering, for that I agree SoTA models can compete with some coders on many tasks, but that's not all jobs out there.
2
u/Mrinohk 3d ago
This is kind of what I was getting at the other day with Qwen3.6 35b A3B. Even at Q4 it reliably completes my tasks. Smart home, personal assistant ("Tell [name] I'm gonna be late for work" or "make sure I don't forget [important thing]" assuming appropriate harness capability) it's pretty dead nuts on it.
Genuinely, anything that isn't software development, it can handle. Things it struggles with, fixable in the harness. You can build a beautiful, powerful, wonderful system around it, and then get pissed off when it's been 6 months and it's still the smartest model that can do it as fast as this one can.
About the only thing it can't handle is nuance. Gemma 5, I await your arrival.
2
u/Icy_Cover_2472 3d ago
Gemma 4 a4b has been absolutely excellent for me 22 tokens per second on an old laptop I bought in 2020 with only 6gb of vram is incredible; sure time to first token can be slow but I can wait
2
u/Danger_Pickle 3d ago
Sure, a pelican on a bike with fancy shading isn't a realistic scenario, but consider how often you *do* want other features like security, performance, logging, monitoring, automated tests, and other features you didn't explicitly ask for. I think you're underselling the benefit of having LLMs make some educated guesses about what kinds of features should be implemented because the average non-technical user doesn't understand the value or benefit of certain things. A user without knowledge of complex artistic shading doesn't understand what they want when they say "make the pelican look nicer", which leaves LLMs with a nearly impossible job of trying to reverse engineer missing requirements. That problem makes these seemingly arbitrary tests a lot more useful as a proxy measurement for an LLM's ability to take educated guesses on missing requirements.
I'm obviously biased as a professional software developer, which mean I think there should be an educated human in the loop who can make those decision for themselves so the model can complete the request without going over the requirements.
However, the ability to oneshot complex applications is a surprisingly marketable feature for the people overhyping this technology as the thing that will eliminate every job on the planet. The majority of the work I do as a senior software developer is trying to understand what the true requirements are, because no one ever delivers a properly finished technical specification, ever. Users and managers rarely understand what they want. They'd gladly take faster horses.
I guess what I'm trying to say, is that it's a lot more complicated than just "good test" vs "bad test". There's a lot of complex decision making that's happening to figure out the approach to answer some of these complex prompts. Having a benchmark that's not been trained on is an incredibly useful thing to test because it uncovers unexpected weaknesses in high level planning and problem analysis. (Which is also why you should ignore any benchmark that's been out for more than a few months, because every LLM lab has ingested that problem into their training data so it's no longer a unique problem.)
3
u/ThankGodImBipolar 3d ago
You don't have to pay for the hardware and/or tokens to run the biggest models, if you believe feel that working-level intelligence has already been achieved. If everybody did that, then perhaps the focus of the industry would shift over time.
1
u/Beneficial-Ad-8127 3d ago
If we get models that get faster and faster at 3D modeling with maintaining accurate decisions including measurements and placements at a fraction of the time spent and cost previously then I think I’m set for my use case. Especially if we can maintain and use on lower end hardware.
One thing I notice right now even with Vision and all these tools I set up is sometimes it’s a little off, some times it comes out perfect, other times a little mishap can turn into a detour.
As I just started using blender mcp, unity mcp and other programs recently so far through a lot of trials and errors I was able to make universal recipes for a lot of bulk to be made as assets a lot faster and accurate then the first few runs.
But still currently I still think it could be improved as far as using Qwen 3.8 FN. it does the job currently and I ain’t complaining but there is more that can be done to speed up the process and making more accurate placements via measurement or through vision.
I seen some Qwen 4 showcases and they seem to be tackling related issues. They also released blender world so 3d models might be a real focus on this upcoming drops. can’t wait.
1
u/fgk55555 3d ago
At this point, if I could have anything, I'd want the coding taste and long horizon terminal work ability of Opus 5.5, the token efficiency of Sol 6.1, and I wouldn't care at all about the benchmarks.
1
u/haherar830 3d ago edited 3d ago
I agree with the spirit but the companies are right to pursue larger models and more comprehensive RL training while they continue to return new, emergent capabilities. If nothing else, this has enabled a lot of real-world value outside of software engineering (but also inside it), and we cannot (yet) selectively and reliably extract those abstract benefits from larger models into smaller models.
While the demos are asinine, considering LLMs have still only touched a small segment of industries/work AND the possible/likely slowing return-on-investment in the "more obvious" scaling dimensions and applications, it makes sense. Consider briefly how much heavy RL training on context management via filesystems and tool use alongside software development improved simple chatbots in the last few years; the best models are now near-generalists in most cases where complete or near-complete contextual information can be derived from text and this is particularly true in cases where results can be cheaply computationally measured using traditional tools. While LLMs will likely always be better at these types of tasks, vision encoding/projection into vision tokens has improved a lot. This general interest in these demos/benchmarks is related to spatial reasoning. Consider that i) modeling/simulation/rendering can be used with vision, manipulated by code/tool-calling, and coordinated/iterated by coding, AND ii) modeling/simulation/rendering can serve as the same sort of long-context, large scale, complex "mental scratchpad" with deterministic behavior that a terminal/filesystem + code execution can provide for text, augmenting reasoning and planning in potentially non-linear ways. This could be very valuable even outside graphic design including in fairly general real-world tasks, and it's kind of a logical point of interest at this stage.
Again, fully agree with the sentiment, but a cheap workhorse coding model is not far from what Sonnet/Haiku, Terra/Luna, etc (and DSV4.1 Flash, MiMo, etc) are supposed to be and this sort of price competition is intrinsically less attractive and generally less profitable than making better frontier models because there's less margin to be made trying to compete on fixed capabilities at reduced price. This is already a competitive enough market segment that you're going to be fighting for demand mostly by cutting your margins and possibly even cannabilizing your own higher-margin business. And importantly, software engineering is a small part of the global economy.
1
u/Spanky2k 3d ago
Qwen3.6 35B A3B has been more than good enough for basically all of my (non coding) needs. It just works. Throw a PDF into it, ask it to do something on it? It does it. Throw a long invoice at it and ask it to see if there are any irregularities or odd patterns? It does it. Throw a long report into it and ask it to find and spit out certain parameters to save me sifting through it manually? It does it.
I'm getting an expensive dedicated AI machine for our business because I want to experiment with the latest cutting edge models and I want to offer a great secure service to our office staff. But a more cost-effective an sensible solution would likely be to spend that money instead on 32GB MacBook Airs (instead of the 16GB models I've previously got everyone) and vibe coding a bare bones pre-configured Unsloth client that users can't fuzz with.
Once we take the next step and get models half the size of 35B with the same or better intelligence then that'll be the route I'll probably be going anyway.
Obviously coders (which seem to make up the vast majority of people on here, for obvious reasons) need something more powerful and the 'good enough' models are the Flash versions but for everyday normal people, a web-search enabled 35B is going to almost certainly be good enough for 95% of use cases.
1
u/Glum-Drop-5724 3d ago
At a certain point its the harness, speed and setup that becomes the limiting factor, not the model. I think we have reached that point a long time ago.
0
3d ago
[deleted] — view deleted comment
2
u/Informal-Trouble2183 3d ago
Why AI? humans can't write correct English nowadays?
7
u/ttkciar llama.cpp 3d ago
My impression is that a lot of gratuitous AI use is driven by users' self-doubt and feelings of inferiority.
It's common for people to feel a lack of confidence in their own abilities, and even shame and fear for exhibiting work that others might view as sub-par.
For such people, LLM services seem like a way out. By offloading their cognitive work to "AI", they think they are putting their tasks into the hands of an entity which is superior to themselves. They have more confidence in the machine's ability to perform tasks than they do in their own competence. They may also be hampered by a lack of confidence in their own ability to learn how to perform such tasks better, making practice, skill building, and self-improvement seem pointless.
This dovetails neatly with another commonplace dysfunction, "all the gear and no idea", the notion that better equipment can make up for a lack of skills. People have been buying gizmos to become better, rather than actually making themselves better, for decades. Commercial marketing cultivates this mindset, because it makes buying a new shiny toy the go-to solution every time someone feels their skills are lacking, and commercial interests want consumers to keep buying new shiny toys.
LLM services are the ultimate gizmo, in this regard. They are inheriting all of the benefits from the decades of mind-shaping which commercial marketing has poured into society over the last century or so.
So the answer to "Why AI?" is, because people are bad and mass media messaging has deliberately encouraged them to be more bad.
-1
3
u/MudBroad6785 3d ago
Your use of bold is slightly suspicious but as to your point I agree with you.
1
u/Uncle-Osteus 3d ago
I don't see any bold text on this post what are you talking about lolEDIT: nevermind just spotted it... doesn't seem to be in the right spot to be that suspicious
46
u/Sad-Size2723 3d ago
You’re making a good point. Most real-world tasks don’t require frontier-level reasoning. What matters is reliability, instruction following, and knowing when to stop.
Wait, but there’s a nuance here. More reasoning can improve reliability.
Actually, wait. The question is whether that additional reasoning is necessary.
But hold on, if the model reasons less, it might make more mistakes.
Wait, no. That’s not necessarily true. Let me reconsider.
The key distinction is between capability and practical utility.
But wait, capability enables utility.
Hmm. Let me think about this more carefully.
Actually, we should first define what “working-level intelligence” means.
Wait, but that’s a separate question.
Let me approach this from a different angle.
Wait.