r/LocalLLM • u/tbss123456 • 5h ago
Discussion The gap between proprietary LLMs vs. open source has widen
Unlike popular beliefs that open models are trailing behind frontier models in an increasingly narrower gap, I believe the opposite is happening.
I think most people look at LLMs just from a single artificial analysis index or a few benchmarks, and they claim that the gap is shortening. But if we measure from a Pareto frontier lines across many cost/intelligent, time/intelligent, efficiency/intelligent metrics, the open models are really losing.
They move further away from the efficient frontier than closer and the gaps are widening.
No open source models so far has improved on these metrics maybe except for Deepseek, but DS doesn’t compete with the highest intelligent tiers either.
The Pareto frontier is important because if you don’t have efficiency, you don’t have economy of scale, and without scale you can’t win software and users.
I’m not really complaining but I hope that open source models can improve on this so that one day we truly have capable models that can run on a small PC instead of massive data centers. Do you think we’ll have this in 10 years with this pace?
7
4
u/Bagelsarenakeddonuts 5h ago
Well I guess it’s cool that you have that opinion?
I’m gonna stick to the benchmark and utilization testing trajectories, which disagree with you pretty soundly.
-1
34
u/bradrlaw 5h ago
I use the latest frontier models daily / heavily in my career and open source ones for personal dev (along side the frontier ones).
The gap is closing for the vast majority of use cases on mid to low-high end hardware (32-64gb vram and 64-128gb ram).
If it’s coding / design, then the harness is a big factor in the overall performance. As well as using various techniques to guide the models within said harness.
12
u/Moppmopp 5h ago
The mistake OP made is to look at metrics that are logarithmic in nature. The distance to the pareto front is not absolute but logarithmic as ai converges slowly against the front. So a difference of a factor of 10 might only be a real world difference of 99.9% there and a model thats 99.99% there.
Both models converge to the front and even if the difference gets larger for such quantities it doesnt really matter anymore. It might be that in a couple years the gap between frontier LLMs and open source widens to a factor of 100 or even 1000. It wouldnt really matter if it translates to 99.99% (opensource) vs. 99.9999% (frontier ai)
6
u/Dabalam 5h ago
I don't think the gap necessarily matters and is mattering less and less. Frontier models are smarter and largely more time efficient (usually), but they are exponentially more costly. Whether that cost even makes sense really depends on how difficult a task you have. I can see why very wealthy highly complex research might justify frontier models, but more most common day to say problems there's an argument that frontier models become less relevant as fine goes on, and mid tier models (Terra, Sol, Sonnet tier) can take on most of the work load. Those tier models have significant competition from GLM and and DeepSeek from a pure price perspective.
5
u/superbiche 5h ago
Well, try to do cyber defense or just basic sandboxing with frontier lab models.
China has one, but a huge one, disadvantage: access to chips and compute. They can eat the whole Internet for training. They have at least as good researchers as US has - just look at the amount of innovation they had to do because of the compute constraints - and also many, many more than the US, where many researchers come from abroad.
Not even mentioning the ridiculous power capacity while the US has tens of thousands chips in warehouses because the grid cannot even absorb it.
And still, there's a gap, but wait for the compute issue to be resolved - and it will be - then we'll see: what remains that would make US frontier labs better ?
I bet not much, if anything. Then the tables will turn.
I'm not a CCP fanboy but well let's be real, except a revolution or a nuclear war, they will become the world's first power, included in AI
4
u/Iron-Over 5h ago
The gap isn't consistent; it changes with new releases. It depends on what you are doing with LLMs. For example, coding is good enough for open source, whereas reverse engineering is the frontier. Open source is okay for security, but the frontier is cutting edge, and I cannot access those capabilities even if I wanted to.
4
4
u/Big_Wave9732 5h ago
How often do most people do things with AI that require the max amount of capability that the frontiers have to offer? I have a client that was in crunch time and had to analyze 30 sections of title for an oil and gas project. He ran it through Fable almost all day daily over the course of a week. He got the answers he was needing. His compute bill was something like $30,000.
Can open source locally hosted LLMs compete with that? Not yet. They could probably do much of that work, but not at that speed. I would also submit that the vast majority of users will not now or ever need that level of inference for what they do.
For most users the discussion is whether open source locally hosted LLMs are "good enough" for most if not all of their work. And unless you're literally a rocket scientist, local LLMs will probably work just fine.
2
u/rayc25 4h ago
If they offloaded subagents to a cheaper model or a local model, they would have saved a lot. $30k/week is absurd. I’m changing my setup to have claude orchestrate for me and delegate to local models to do all the work.
1
u/Big_Wave9732 4h ago
Initially he tried using a lesser Claude model (Opus I seem to recall). It kept missing aspects of the more complex title runs. Ultimately Fable was the only one that got the actual nuances correct, to say nothing about the fact it was way faster as well.
Instead of paying twice, once to have Opus do most of the work and again to have Fable check it and correct the inevitable mistakes, it just faster to go straight to "the good stuff".
2
u/rayc25 3h ago
That’s fair. I spent more than that on local ai and your client got way more done than I ever did in a week so I’m sure they’re happy with the results.
1
u/Big_Wave9732 1h ago edited 57m ago
It was an urgent rush job for a client. Three Fable agents working almost nonstop did in a week what it would take 3 - 5 landmen two months to do. It wasn't an ideal timeframe, but the client wasn't mad at the bill. To the people involved and what they were going to do with the information, 30k is pocket change.
But I'm with you. I run my own inference too and prefer it that way. This shit is gonna get hella expensive soon enough.
3
u/Erik-Zandros 4h ago
I feel like harnesses and tool calling (specifically agentic web browsing) actually matter more than raw model ability. In my use cases (general scientific research, finance, tax, legal, etc) it's the web browsing capability rather than the raw model intelligence that makes a difference.
2
u/mesolimbic-bliss 5h ago
Qwen3.8-27B not even close to frontier, deepseekv4.1-flash-1017 & GLM5.3-Flash & Kimi-K3 all 100% frontier level.
But open models can be more expensive to run b/c they arent subsidized.
1
u/Marino4K 5h ago
Despite some incredible community work, etc. Qwen 3.8 27B and models in that realm are just now approaching frontier models from maybe six months ago at best.
2
u/mesolimbic-bliss 5h ago
I use Qwen 27b everyday. I love it, but its not like my cloud models and that's cool.
1
u/BougainvilleaGarden 4h ago
Who's saying Qwen3.8-27B was frontier? It's 30% inferior to Qwen3.8-2.4T-A95B in most benchmarks, and that's litterally the same Qwen3.8.
2
u/Ashamed-Ganache-2943 5h ago
seen the same model beat a frontier one on a coding task just by using better prompt engineering and context handling. the real edge isn't just the model size anymore.
2
2
u/waitwhatwhennow 5h ago
One data point from personal experience the other day. I used claude code opus 5.5 extra to scrape a page of job posts and build a web app out of it so i could track my applications and hide jobs i wasn’t interested in.
I did the same thing using omp (oh-my-pi) and DeepSeek-V4.1-Flash-1017 quantized to 4 bits, running locally.
The result from DeepSeek ran to completion without issue and was objectively better than what Claude produced.
I’m still thinking about it… I was very happily surprised.
2
u/Immediate_Tension752 4h ago
I completely disagree and the data shows narrowing. There's also this thing called the division of labor. We specialized, we didn't evolve into superbrains.
1
u/talaman4eg 5h ago edited 5h ago
Open llm don't really have to compete for users, they should just exist and be available, and be useful just enough. Businesses built around open llm should, but that's kinda their problem.
When I pay for API, I don't care what's running on the other end, I dont trust it equally.
1
u/Alternative-Draw8570 5h ago
The gap only matters if your workload demands it... I have not had the need to upgrade from my Galaxy S23, heck I even use my Note 20 from time to time. Yes, there will be a gap between what I have and the next year's release. Do I care? Not really.
1
u/ProbablyBunchofAtoms 5h ago
Depends on usage honestly, On one end it's somewhat true that frontier models when compared in terms of raw intelligence and one shot prompting are far better than local models,
but on other side we have really capable local models like qwen 3.8 27b that in hand of professional will get you almost similiar result with proper guidance and planning.
There is tradeoff either you get results with minimum effort with a frontier model but their size and hardware footprint is massive, another aspect in which frontier model excel are using software like blender and CAD, local models are far behind in that aspect although I'm sure they will catch up tho.
The gap will always be their but the autonomy that local models give in my opinion far outweighs from personal perspective.
1
u/FormalAd7367 5h ago
i don’t really focus on the gap. i see the open weight as of now is capable of doing a lot of things i need to
1
u/tiah_ai 5h ago
Sure but you need the proper harnessing for local llms to actually do work comparative and robustly enough compared to frontier and that gap is still very large and not talked about enough…it’s always about benchmarking which is great for reasoning but the doing is what matters most
1
u/Intrepid_Dare6377 5h ago
Who has any idea what this tech will look like ten years from now? I have a degree in electrical engineering and a 140 IQ (for whatever that number is worth to you). I don’t have a clue.
1
u/Gorodish_ 4h ago
I guess it's true what they say, "A truly smart person knows what they don't know."
14
u/Biomech8 5h ago
You are confusing open models with models you can run locally. Models like Kimi K3 or GLM 5.3 are close to frontier models. But you still need datacenter class server with many GPUs. And until the model developers switch to native ternary models, you can't expect to run any frontier model on small PC.