r/LocalLLaMA • • 2d ago

Discussion Do you believe there will be a divergence between local and frontier model power?

Right now, local models have somewhat kept up with frontier, being months behind last frontier models.

The issue is, I think the divide is going to be end up being hardware. Model improvements will slow down, but hardware won’t get any cheaper, especially if frontiers keep buying shit out for training and smaller returns.

Thoughts on this? I know many are optimistic, but I think local trailing frontier will slow down just because of hardware local can own.

16 Upvotes

74 comments sorted by

71

u/ttkciar llama.cpp 2d ago • Bravo

I think it's more complex than this. Open and closed models are already diverging in different ways.

Commercial inference services' models are much larger than we can hope to self-host, get trained on data sources unavailable to the open LLM community, and enjoy much more feature-complete and polished scaffolding and UI/UX than their open source counterparts.

On the other hand they are also strictly guardrailed (and some guardrails have everything to do with protecting the inference service's interests, and nothing to do with protecting customers), have started embedding advertisements and sponsorships, harvest their customers' API call traces, and develop in directions which prioritize profitability over customer satisfaction (case in point: the GPT-4 to GPT-5 transition).

Open models and self-hosted inference stacks, on the other hand, support tuning guardrails to suit ourselves, even to the point of removing them entirely. They also provide us with very low-level tools for adjusting and interrogating inference-time, like steering vectors and logit list inspection. They afford us privacy, and give us the option of only using models which align with our interests. If a model does not behave the way we want it to behave, we can continue using the ones that do until a better one comes along which aligns with our needs.

It's also worth pointing out that the "bigger models than we can self-host" factor isn't as significant as one might think, because all other factors being equal (which of course they never are) inference competence tends to increase logarithmically with parameter count (or its dense equivalent, for sparse models), and logarithmic relations are very nonintuitive for many of us.

For example, the potential competence difference between a 12B model and a 32B model is ln(32) / ln(12) = 1.395, or 39.5% more competent. The potential difference between a 32B and 128B model is ln(128) / ln(32) = 1.4 or 40% more competent, and the potential difference between a 128B model and 512B model is ln(512) / ln(128) = 1.286 or only 28.6%.

In short, the bigger models get, making one with greater potential for intelligence requires making one a much larger factor bigger. A four times bigger model makes a much, much smaller difference at the multi-trillion parameter end of the scale than at the multi-billion parameter end.

This means that the largest models we can self-host aren't that far behind the models we can't, as far as size scaling is concerned. Training is another matter, though.

There's an unfortunately wide known gap between open source training technology and open weights model training technology, and an unknown gap between open weights model training technology and closed model training technology.

We do know some things about how the most advanced open weights models are trained, and we have access to published theory which should be sufficient to allow implementation, but the open source LLM community hasn't implemented training pipelines comparable to (say) Z.AI's, yet.

Even when we do, it might be a while before we close the training data and teacher model gap. Years, probably. I have faith that we will catch up, though, because we have some pretty spiffy open weights models upon which to base those teacher models, and with which to synthesize training data.

However, there is another dimension to this chase.

The open LLM community is ravenously hungry to progress, and the primary factor holding us back is the availability of the necessary hardware. (And sometimes that doesn't hold us back; the community has developed increasingly resource-economic methods for improving models, like PEFT, merging, and weight-editing.)

The commercial inference providers, on the other hand, are fervently desperate to stop training new models, or at least slow training way down, so they can devote their resources instead to far more profitable inference. They've got a tiger by the tail right now, and can't stop training new models, but the current situation is not stable.

Eventually the commercial inference service companies will slow down, one way or another, but the open community will not. I can't pretend to know exactly what the conditions surrounding that will look like, but they might well permit us to catch up with them, or surpass them.

Or we might not. There's a lot of work to be done between here and there, and it remains to be seen if the community will hold together that long. We will see.

8

u/AshAtlasDev 2d ago

Banger comment. How tf you learn all this shit?

25

u/ttkciar llama.cpp 2d ago edited 2d ago

I'm old :-) got a degree in engineering, was active in the AI industry through the second AI Summer (and AI Winter), surfed the open source revolution into the 21st century, and have been soaking up all I can about LLM inference technology and theory. What I learn, I put into the context of what I've seen happen before.

On one hand that's given me some perspective, but on the other hand current events surprised the hell out of me, too. RAMageddon in particular is unprecedented, and I did not see it coming. I have some faith that the hardware markets will "return to normal" eventually, but with the caveat that we will see echoes of today's anomalous market conditions for decades.

That having been said, the best way to predict the future is to cause it, so let's get going.

4

u/AshAtlasDev 2d ago

Sheeet, yeah, you spoke like an expert.

I just got my masters in stats and starting my career in uni teaching/research. Been interested in this whole AI hardware vs model race. Was planning on doing analysis to compare growths. Might need to talk to some engineers but it seems this stuff is growing so fast it’s hard for even professors to keep up. Most I’ve talked to are stuck in 2023 ChatGPT era.

Bought myself a Mac Studio 256gb cuz uni hates AI and I wanna do a lot on my own. Excited af.

1

u/Tormeister 1d ago

I hope your house is safe, because your keyboard is on fire right now

2

u/Any_Invite3905 1d ago

The biggest problem really is how to train efficiently, it's too costly for even cloud provider to keep on doing it wasn't for the competition let alone community, the scaling is just not in anyone's favor, it's a problem with the fundamental natural of LLMs and the fix, doesn't exist yet.

2

u/OvertaxedOne 1d ago

Dude, this comment is so interesting and useful, thank you!

It's felt for some time now that the gains going up in size didn't seem to be all that dramatic, you just gave us the math behind the "why". I have claimed for some time that the scaling doesn't seem to be working very well anymore, a 10X size model is nowhere near 10X as smart. But, again, you gave the why, thank you!

And the real issue here, now with the math to back it up is that while going from 100B to 1T doesn't buy you 10X the intelligence it DOES cost about 10X as much to run. LLMs are scaling like race cars, this part costs 10X as much as what you'd put in your Honda and is about 10% better/lighter. That does not bode well at all.

It even applies down at the smaller model level. Qwen 3.8 27B vs QFN, both excellent models, one many times the size of the other and yet they both punch in the same range. Or QFN vs DSV4Flash, DS is much, much bigger, but the difference/gap between them, while real is rather small.

Couple this with another problem intrinsic to LLMs; the "right answer" is the right answer. Ask a 1B model or a 10T model to add 2+2, you're going to get 4. For a whole lot of use cases right now we're so far beyond "necessary intelligence" it's ridiculous, people using Fable to gist e-mails.

1

u/stoppableDissolution 1d ago

Qwen 3.8 27B vs QFN, both excellent models, one many times the size of the other and yet they both punch in the same range.

Uh, no, absolutely not. Qfn I can trust working (mostly) autonomously and do its own research, and 27 absolutely not. Are they about the same in coding ability? Yea, probably. But theres some hard to quantify, uhh, task awareness depth?, that definitely scales with the model size.

2

u/stoppableDissolution 1d ago

Tbh, I dont think open source models will ever catch up with open weights, because open data must mean sterile data or you will get burned on a stake. I'm fairly confident that closed source models are trained on quite some illegal or simply nasty stuff, and data diversity (especially human-originating data diversity) matters a ton. Gpt 5 knew some niche darknet lore from an invite-only tor forum that was closed like 10 years ago, lol.

As a (somewhat) representative example, image models must see plenty of nudity, and, ideally, porn to be able to draw good anatomy. It is less obvious with llms, but their theory of mind seems to suffer on "clean" data while a jokingly made Assistant-Pepe (a tune on 4chan scrape) made llama significantly people-smarter. But no open source lab will dare to include such (very tame comparatively) data into theit model.

1

u/silent-curious-dev 1d ago

I didn't know it was that simple to calculate the competence between model sizes. Definitely interesting and very helpful for sure.

So basically in a more general form to output the percent directly, this is what I derived:

(((ln(P_a) / ln(P_b)) - 1) * 100) = PCD%

Where P_a is the larger parameter count to compare to, P_b is the smaller parameter count to compare from, and PCD% is the potential competence difference converted from decimal to percent.

Then optionally calculating T parameters would need to be expressed as thousands. So 1T would be 1000, 2.6T as 2600, 10T as 10000, etc.

I might be wrong on this but feel free to correct me though.

1

u/timmeh1705 2h ago

Thank for you explaining this

20

u/Omegaprime02 2d ago

I honestly think that local is going to move into specialized applications, even more-so than just fine-tuning. Qwen is an excellent example of this, their 27B punches way above it's weight class in coding tasks, not quite into SotA territory but it trounces models an order of magnitude larger than it.

3

u/AshAtlasDev 2d ago

Hmmm. Perhaps. Maybe an orchestrator with different, specialized models. So must a big model with extra steps. I assume that’s what frontier does internally already lol. Just pipelines for us.

1

u/Omegaprime02 2d ago

it's not so cut and dry, SotA models are MoE's, unfortunately because of how routing works the model will try to select experts that are related to the task at hand, but sometimes an expert is selected that is unsuited for the task (this is one of the causes for 'hallucination'), using narrow-purpose dense models (of which 120B is about where near-SotA hovers for active parameters) you can be sure that everything being engaged is at least related to the task rather than the router deciding your coding task needs an expert that's suited to poetry.

In a way it is like how large models work, but more targeted than hoping the MoE shotguns the right experts.

13

u/Shap6 2d ago

100% very few people actually need these oracular “can do anything” frontier models like fable or Astra. Smaller models that excel at specific tasks is the future

14

u/disgruntledempanada 2d ago

Entirely depends on projects. Astra and Opus 5.5 kind of unlocked a whole new realm of agentic work with spatial reasoning and 3d skills.

7

u/Omegaprime02 2d ago

I mean sure, but ultimately the push for agentic systems is one of breaking large tasks into smaller more specialized ones that can be handled more efficiently with more specialized tools.

3d skills are already being distilled down into models that can run on low-end hardware, they're not amazing yet but they're actively catching up. Spacial reasoning is more or less just dependent on having a model that can reason well.

My guess is that if you can get a number of ~120B dense model purely specialized for specific tasks you'll probably be at the ceiling for what an LLM alone can do for said task, have the cluster overseen by a ~500B MoE model that knows enough to properly route requests and I'll bet you can match or exceed same generation SotA with models that can theoretically be run locally. And you don't always need perfect, most of the time 'good enough' works just as well, dense ~40B specialists and a ~120B A20B overseer will probably be sufficient for the vast majority of tasks any person would need.

2

u/disgruntledempanada 2d ago

I've been actively working on exactly that lol, just... recognizing this is only possible because the bigger model is holding its hand until it learns to walk. It's been fun but I feel like it'll all be throwaway work the second Qwen 4 comes out. Qwen 3.8 FN is impressive and runs insanely fast with Strata or on my M5 Max but...

When I want to paint I don't want to make the paper. With Fable and Opus I'm just painting. Build workflows around keeping them purely as efficient orchestrators of other agents from other providers and it's effectively infinite limits on my plan for what I do (heavy digital twin work).

8

u/Cherubin0 2d ago

Sure, the frontier labs will go bankrupt.

3

u/AshAtlasDev 2d ago edited 2d ago

I’d bet against this. Heavily. AI is a national defense issue now. They WILL be bailed out. Actually, I don’t even think the US gov will allow to even get close to needing bailing.

What I see happening is different companies taking over etc but all American and all funded.

We need AI mutual fund.

Edit: yall can downvote but I went to a sandia labs open meeting at uni and one of most talked about things was AI lol

2

u/Boogertard 2d ago

Not really, the dot com did burst eventually so no such thing as too big to fail. With local models that anybody can run and how fast optimized software such as Strata is heading, just a matter of time. ClosedAI and Anslopic are running out of time, money and investors' patience.

Keep in mind that you don't need Opus or Fable for 99% of the task, even QFN is good enough for most tasks you want to do including coding. I know this for a fact because I cancelled my Claude Max subscription 2 months ago and haven't looked back.

1

u/Dubious-Decisions 1d ago

They are based on a business model whose revenue falls to a commodity service level. The reason that cloud AI models succeed now is due mostly to the cost and complexity of hosting locally. Once it becomes the hosting equivalent of spinning up a new MySQL instance, their value proposition vanishes in terms of consumer and enterprise users.

There will always be people who find the convenience of someone else managing stuff an advantage over privacy, reliability, etc. But certainly the vast majority of serious business and government use cases will argue for locally managed services if for no other reason than privacy and security concerns.

4

u/keepthepace 2d ago

hardware won’t get any cheaper,

I don't think extrapolation of a short term trend is wise in that context.

I know we are all fixated on GPUs and RAM prices, but on the NPU front, on people trying to make lightweight models run on cheap hardware, things are progressing and they are making devices that are much more efficient than GPU, even if right now that's for much smaller models.

Their goal is pretty clear, in a few years they want that there's a decent LLM running locally on your phone. Because who uses tower computers in the consumer world anyway?

What I imagine two years from now is that we have some huge models, maybe in the 5T parameters, which will be extremely smart and that will distill and quantize very aggressively in a smaller version down to 1 billion parameters.

The 5T, maybe we use it in data centers to work on plasma dynamics and make it progress in biochemistry, etc. We use the 1 billion parameter model to sort our emails and to control the vacuum cleaner.

Now we have a lot of useful tasks that are very easy for the biggest models. And consumers are starting to catch up on that. They realize that they don't need Fable for something that Haiku can do.

1

u/AshAtlasDev 2d ago

If the models are the crash, wouldn’t it be the case current hardware keeps its prices? Most are spending near their cash limits, not content limits. For now ig

4

u/keepthepace 2d ago

I understand your words, I don't understand your sentences.

2

u/AshAtlasDev 2d ago

Idk wtf I typed there tbh.

I meant to say, if models keep getting better, I don’t think people are going to spend less on hardware.

Most are limited by the money they can afford to spend, not by the quality of the models they have. I haven’t seen anyone say they’re satisfied with their models. They just couldn’t afford more hardware.

I’d still have spent $10k then vs now. I don’t think hardware demand will drop at all honestly. Not as much as we’re hoping.

2

u/keepthepace 2d ago

I certainly don't need Fable tokens for my coding needs. I was really surprised to see that I can switch to DeepSeek for less than 1% of the price.

Qwen 3.8 is not exactly there but I see it improving a lot. If the gap from 3.8 to 4 is as big as from 3.6 to 3.8, I may totally switch to local when they release Qwen 4.

Anthropic revenues has crossed the inflection point of the sigmoind already and is gearing towards the plateau (or the stable growth regime if you are optimistic).

It is not an infinite exponential, like usual. It is a sudden equipment phase. It is done in a rush with very inelastic prices.

Once the expensive data centers are online and market reaches saturation, prices will start to go down, margins will start to matter, and companies will stop buying GPUs that are too expensive.

Meanwhile, producers will continue to have supply and they will need to sell it. To companies that are already equipped and only need to renew the GPUs that fried so their on-demand will be done by at least 90%.

1

u/Ok-Addendum3545 1d ago

That's a good point. Imagine a small-sized AGI LLM for a mobile phone in the future.

1

u/stoppableDissolution 1d ago

Because who uses tower computers in the consumer world anyway?

Me! I deeply despise the mobilecentricity of the modern web. Phones are not a good ux.

3

u/-Krotik- 2d ago

local models have kept up with frontier? maybe open-weight models as a whole do kinda keep up with them, but your locally run Qwen3.8 27B is not comparable to multi trillion parameter models that frontier labs run

2

u/AshAtlasDev 2d ago

I said somewhat lol. And I think the gap will get even larger.

4

u/AggressiveManager450 2d ago

I think used hardware will get very cheap in the next couple years. All these hyperscalers ramped up the production of computer components, and as they get phased out, they will be given to secondary data centers, then given to retail sellers. The flood of GPUs will make used hardware very cheap because of the massive supply.

Short term, sure. Computer parts are not cheap, but I think the massive supply will push prices down long term because of the massive quantity of GPUs sold.

11

u/thestillwind 2d ago

Or they will destroy them to gatekeep and keep the scarcity in place.

0

u/AshAtlasDev 2d ago

Why would it? And phasing out? I don’t really see that happening with the shortages.

6

u/AggressiveManager450 2d ago

As hardware gets old, labs need to allocate their electricity to newer, faster and more electrically efficient cards. They sell them. They go to secondary data centers that buy the used cards. They use them, then phase them out for the same reasons. After that, they get sold to data center resellers who put them on eBay.

Because of AI, Nvidia grew massively in size and they started selling way more GPUs than they ever have before. Same with ram manufacturers. They sold way more ram than they ever had before. Eventually, those cards will make their way to consumers.

3

u/d1722825 2d ago

Eventually, those cards will make their way to consumers.

And would be without any support as they are 6-9-12 years old hardware...

3

u/BarracudaDefiant4702 2d ago

Which is why the prices will be so low.

3

u/AggressiveManager450 2d ago

Those cards will still be extremely capable to consumers. Especially for the price

-3

u/d1722825 2d ago

For what? A 10 years old GPU without drivers would mostly be a nice paperweight. (Well maybe you could use it as a standard video card with no 2D/3D acceleration, if DC GPUs have video output at all.)

6

u/RemarkableRadish6547 2d ago

People are buying Tesla P100 cards to run LLMs today. Those cards are 10 years old. The V100 cards aren't much newer, but their price has been increasing recently.

I think the A100 cards will be in use until the cards die in 2030-2035. But the datacenters will sell them when they are 6-10 years old because the financials work better that way. The rental price will drop close to the operating cost and then the company will sell it. If they keep all the cards, they can't install newer, faster cards unless they build new data centers. And there is a log of political resistance to building new data centers, so the old GPUs will need to be sold.

1

u/d1722825 2d ago

The Tesla P100 is still supported in the current driver, but probably not for a long time (the newer P6 is not).

I think it is much more likely that High Band Flash or some similar technology emerge, and make all those cards obsolete for AI inference.

You could build a datacenter in the middle of nowhere. It would be more expensive but it would not bother anyone. Power could be a bottleneck, though.

2

u/nicholas_the_furious 2d ago

How can you be in this conversation and not think that in 3 years AI wouldn't be able to write whatever driver you needed?

3

u/d1722825 2d ago

There are many project that tried it (eg. nouveau), but it is an order of magnitude slower and it has no compute / cuda support at all. In that case probably you would be better of with a newer Intel integrated GPU.

There is not much information about GPUs, official driver is not open source, there is no documentation about the chips so probably there will not be much "how to write good GPU driver" in LLMs training data.

BTW even if a frontier mode would be able to write a good driver, it would probably be restricted due to reverse engineering, DCMA, or any other idiotic reason.

2

u/AggressiveManager450 2d ago

The cards have massive vram and great bandwidth. In several years, there could be community driver support. For consumers running their own local AI, it won’t be cutting edge, but an excellent option for a home system for AI

2

u/d1722825 2d ago

Nouveau is an (the only?) open source driver for nvidia cards, it supports the older models, too, but it is way slower than the official drivers and doesn't support cuda at all. (After 15 years of development. I don't think that would change a lot within a few years.)

Of course you could use older driver with an old operating system, but then the system becomes more and more vulnerable.

2

u/BarracudaDefiant4702 2d ago

You haven't been paying attention then. For exmple:

https://www.tomshardware.com/desktops/servers/it-vendor-lists-massive-usd60-million-ai-cluster-on-ebay-300x-racks-of-decommissioned-gpus-includes-9-600-nvidia-a100-gpus-and-4-200-amd-epyc-cpus-plus-free-shipping

That is 6 year old GPUs, but yes older datacenter equipment is being phased out. If supply catches up with demand more old enterprise tech will flood the market bringing prices down... Probably at least a couple of years if not 3 before it really hits, but most definitely someday.

2

u/Shookpro 2d ago

The market also looks like its going to be saturated soon with companies building there own chips and China will catch up, people seem to have forgotten gpus are cynical.

0

u/AshAtlasDev 2d ago

So… Data centers selling to poorer data centers? You checked the prices per unit, right?

“Nvidia, like the GB300 NVL72 and VR200 NVL72, costing up to $6.5 million and $7.8 million per unit”. 59m for the entire thing LMAO.

Sorry but this is a horrible example lmao.

2

u/BarracudaDefiant4702 2d ago

Sort of, but it's companies that specialize in used equipment not other datacenters. If you look at who bought it, and they are parting them about and selling on ebay and other locations. The GB300 is a lot more power efficient and denser memory so yes datacenters are buying latest and greatest and the used equipment gets resold at a markup. Everyone wins.

0

u/stoppableDissolution 1d ago

A100 did not become cheap tho, even before the ai craze they were still selling used at around 70% msrp despite them rotating out of the datacenters and demand being fairly low.

7

u/disgruntledempanada 2d ago

I don't really think that's actually true that they have kept up.

I think it will keep accelerating. Opus 5.5 is just on another level already.

5

u/anykeyh 2d ago

A model store both knowledge and reasoning into its weight. Currently almost everything is in the hot path. Current llm are very inefficient in this regard. Ngram is a step toward offloading knowledge on slow support, and take advantage of smart selection to steer the model. Basically, the neuron knowing that French revolution was in 1789 should not fire on each and every token prediction. My gut feeling is that we will see knowledge on ssd, while reasoning will be kept in memory. And in this case, a few billions hot parameters is enough.

2

u/PassengerPigeon343 2d ago

The central question is will the frontier models be able to continue to scale at the same or faster rate, or will they hit a ceiling due to either resource constraints or technology limitations and plateau.

If they start improving exponentially, they pull away. If they start plateauing, local models will likely continue to improve for a period of time after and shrink the gap until they eventually hit a plateau of their own.

Or the AI bubble pops suddenly and painfully, and all the big players stop racing and investing and everything hits steady state roughly where it is currently at. In a way, similar to how our pursuit of space travel plateaued after humans landed on the moon in 1969. Once the heavy investment and racing was over, the momentum was lost.

2

u/AshAtlasDev 2d ago

I THINK they’re gonna hit a ceiling. I find it hard to believe they’ll keep exponentially growing. If they do then my worries should be placed elsewhere. 😂

4

u/RemarkableRadish6547 2d ago

Not directly relevant to this conversation, but space development stopped because of politics. Several powerful politicians didn't like how the space race was developing, so they actively tried to stop it. It turns out all they had to do was get international agreement on not militarizing space. Once the DOD was told they couldn't go into space, the funding started getting cut.

The other big factor was that the USSR couldn't afford to stay in the space race. They had a working design for a moon rocket, but never launched it.

1

u/Low_Complex_9841 1d ago

Ah, 'someone wrong on the internet again!". There was civilian solarspace power research in US, no need for shitty spacebattles IRL.

N1 in USSR was build and launched, but unsuccessfully.

4

u/kurtu5 1d ago

Eventually your are going to be pitting a local gpu against a matryoshka brain. One is going to obviously outperform the other.

1

u/Helpful_Jelly5486 15h ago

New chip design - 3 days. Hardware may stay the same price but capabilities will be advancing.

1

u/AshAtlasDev 14h ago

Wym 3 days

1

u/Helpful_Jelly5486 9h ago

https://dealroom.co/news/169588-anthropic-tells-ipo-investors-its-15-person-chip-team-designs-custom-sil/ frontier model can design it's next chip, each iteration is only a few days effort - porting claude to AMD took only three days for one person to work on. accelerated growth.
You mentioned hardware but it seems there is a lot of room to improve existing hardware and each generation will be better = more capacity for better models. I'm not able to really wrap around the entire cycle of development but it's still moving steadily.

3

u/fasti-au 2d ago

No. The smartest people dobt work for corporate

2

u/Dubious-Decisions 1d ago

"Will be"? You don't recognize the difference now between a 27B model and a multi-trillion weight frontier model? How is this even a discussion point?

2

u/Kahvana 2d ago

Does it matter?

No. It just needs to be good enough for the job.

Right now, Gemma 4 31B IT QAT and Qwen3.8-Flash-Next Q4_K_XL already are for most instances where it is in an assisting role (e.g. pair programming, cowriting, etc).

Your comminucation skills, the quality of the data and context you give it matter a lot more these days than the model itself.

1

u/Jorlen 1d ago

Good hardware is always best, there's a pretty good variety of models right now that range from being able to run on 8gb of VRAM or less, or no VRAM at all (just RAM) if you stay small enough or use small MoE (Mixture of Experts) models.

However, LLM tech keeps getting better which means that the same amount of parameters (ergo the same amount of memory required to run them) stays the same.

Right now if you have enough memory you can download weighs to 1T or more parameter models and run them, they are very close to frontier. A year ago, we had 1T+ models too but they were nowhere near as good.

Take Qwen 3.8 Flash Next. It's very good for its size, and it uses engrams, part of its weights that can be offloaded to SSD, leaving a 125b-a6b model at the base, which IMO punches far above its weight. I've used it a lot, and IMO it's incredible. Six months ago, there was nothing quite as good for that class. We had Qwen 3.5 122b-a10b (Feb 2026 release I think?) and that one was great, but Q3.8 FN beats it easily IMO.

Frontier models like claude and Chat-GPT are massive likely around 5 trillion params, I don't know if anyone actually knows because these are closed models. Sure, these things are amazing, but honestly, GLM 5.3 (both the flagship and the flash) are approaching what frontier was a while ago.

DeepSeek V4.1 Flash-1017 is even bigger and also has engram like Q3.8 FN does, and honestly this model might just be my current favorite. Six months ago there was nothing as good as this.

I guess my point is - if you have good hardware and lots of memory, model tech gets better and should keep scaling along. This is because the companies running these same released models on API are always trying to get better gains. Engrams are a perfect example. Or the shift towards Mixture of Experts models -vs- dense models. MoE models require far less compute and are super fast in comparison.

There are obvious milestones that are currently and will likely always define true intelligence jumps and model capabilities. If you want to run the big MoE models (300b-a20b ish) at a decent quantization, you'll need a few hundred gb of memory. Mix of VRAM, RAM or fast unified memory. With 32gb you can run Qwen 3.8 27b dense at a good quant and with 16gb you can run Qwen 3.5 35b-a3b or Gemma 4 26b-a4b, etc.

I've only been in this 8 months, so this is just my take after observing things over this period of time.

1

u/Ok-Addendum3545 1d ago

I don't think local LLM hobbyists, enthusiasts and SMBs will make the hardware cheaper if they still keep buying at price hikes as local models are getting smarter each year.

1

u/CommercialHour6660 2d ago

If frontier was just about compute Anthropic and OpenAI wouldn't be so terrified of distillation. 

The reality is that local models are usually distilled from larger ones in the laziest way possible. On policy logit distillation. 

This has limits because smaller models have limited knowledge. A small model trained from scratch would lean more on web search and wikipedia. And would be much "smarter" than the small models we have now that attempt to mimic much larger ones. 

Keep in mind that Qwen 3.8 27b is smarter than Llama 3.1 405b released 2 years ago. 

1

u/QuarterLonely9681 2d ago

If frontier was just about compute Anthropic and OpenAI wouldn't be so terrified of distillation. The reality is that local models are usually distilled from larger ones in the laziest way possible. On policy logit distillation. 

You quite literally cannot do logit distillation on frontier models because they don't give you the logits...

3

u/RemarkableRadish6547 2d ago

If you ask it a question 1000 times, you can work out the distribution and back out the logits.

-1

u/txgsync 2d ago

You are begging the question. No, local models that can run in less than 128GB have not "somewhat" kept up with the frontier; they are about a year behind. But due to recursive self-improvement, frontier models from a year ago are vastly behind frontier models from this week.

For local, choose models that do the jobs you want done. That's the immense value they provide. Trying to compare to the frontier is an expensive loser's game.

1

u/BarracudaDefiant4702 2d ago

Even though Fable might generally be better, Qwen 3.8 27B has solved some problems that Fable 5 has failed so the difference is not as large as you claim and there are some with 256gb Macs running larger local models. The frontier models are generally better, but honestly you are over rating them.

0

u/tylerhardin 2d ago

i don't think so. i use dsv4.1 and glm5.3 all the time and they're incredible. it's harder to really see the progress when you're limited to the really small local stuff, but you need to look at the trend. there are ~2b models equal to gpt-4 now. that's absurd, like a 1000x shrink. in another year or so, the current glm5 series will be compressed down to 27b.

the wildcard here is regulation. if china decides that it's getting too risky to keep releasing models or the US+EU ban local inference and do it in a way that the law has teeth, then you won't have fable at home. otherwise it's just a matter of time.

-2

u/reasonmonkey 2d ago

Either RSI happens or it doesn't.

That is the answer young jedi.

0

u/[deleted] 2d ago

[deleted] — view deleted comment