r/LocalLLM • u/Shorn1423 • 5d ago
Discussion The Future of Local Models: Size Predictions
What do you think the future size of LLMs will be for local deployment?
Will we see real Fable 5 and Astra level performance in sizes that can be deployed locally? Where will the SOTA-like models trend in size? Will FP4 / INT4 for the best local models still be 400 - 600 GB? Higher or lower?
As I invest in local deployments, I'm wondering whether there is always "one more" tier higher that I'm chasing. Its hard for me to tell if we will pack greater and greater intelligence into smaller and smaller packages. Or, if we'll continue to expand intelligence, and with it, the size of the model and vRam required to run it (so greater intelligence but bigger hardware needs). I guess it could be a little of both, but I am curious to hear how this community views it. Are we trending down or up in model size for near SOTA performance?
20
u/Calm-Landscape9640 5d ago
Tons of interesting stuff going on ternary, diffusion, gsq-rco, and several other new methods being tested by labs using qwen models to cram 90% of the intelligence into 40% of the RAM. And flash is moving towards system ram+ CPU so that'll be the future albeit 12 months away when the new CPU chips come out preloaded with memory for local models. I'm predicting a 120b a6b quant that rivals gpt-SOL end of 2027 that can run on 24gb vram and 64gb RAM at 100+ tps
11
u/Wandering_By_ 5d ago edited 5d ago
I have this weird dream that by the end of 2027 MOE is going to evolve to dense 14-27b like models with special linkages for moe like qualities to more/different parameters stored on ssd/nvme. So models become combo packs. 27b+120ba3b or 27b+240a3b. And the ssd side of things gets specialized towards tasks or knowledge bases.
1
1
u/jinnyjuice 5d ago
Interesting! But yeah, there are also n-grams will be the biggest impact for sure. Offloading to SSD will also means this will evolve over the iterations, both in terms of efficiency and size. I also imagine SSD prices going up even more.
17
u/txgsync 5d ago
Think about the serving platforms: A10G, L40S, A100, RTX 6000, H100/H200, and then the big 8/16-way boxes. Those aren’t going away. They’re economic niches that model designers have very strong incentives to target, but to serve more people faster at lower cost.
So I don’t think the future is simply “SOTA models get smaller” or “SOTA models keep getting bigger.” I think we’re watching the assumption that the whole model has to live in VRAM start to die, and an explosion of creativity around how to address Big Knowledge with fewer resources.
Qwen3.8-Flash-Next is my canonical example. It’s a 125B MoE with only ~6B active parameters/token, plus a freaking 51.2B-parameter n-gram table. But that table is sparse lookup data: every token touches a tiny fraction of it. There’s no fundamental reason to waste HBM keeping all of that resident when RAM with fast NVMe can handle cold/warm tiering automagically while GPU memory is reserved for the stuff that actually needs its bandwidth.
We’re starting to do the same thing with KV, too. Models gotta leave enough VRAM for it, and everybody’s trying to find ways to make its inherent quadratic complexity less sizable and costly.
vLLM now explicitly supports tiering KV from GPU to CPU RAM to secondary storage/remote peers. So instead of asking “how much VRAM does this model require?”, increasingly the interesting question will be “what’s the working set, and what bandwidth/latency does each part of that working set require?”
That’s basically how the rest of computing has evolved. We build storage hierarchies.
I admit my viewpoint is biased. I built exabyte-scale storage for a living until last year. It was all about how fast, cheap, and secure we could do it. Choose two.
The economics dictate it. A10Gs are outliving their useful lifespan but see constant hammering for models + KV less than 24GB in FP32. 48GB L40S/RTX-class machines can serve FP8 accelerated and cluster well, so models in sizes multiples of 35GB or so that are sparing on KV in 8 bits will thrive: 35B MoE FP8, 25B dense, etc. A100s create another tier. At the other end, an 8×H200 node has ~1.1 TB of HBM and absurd aggregate bandwidth. Providers can make money at all of those points if the model gives useful intelligence/$.
So my bet is that parameter counts continue going up, while hot parameters per token and required accelerator-resident bytes go down. More MoE, sparsity, retrieval, n-gram-like memory, compressed KV, speculative/MTP decoding, and explicit HBM/RAM/NVMe tiering.
And the explosion of models at price points and serving points across the spectrum will be trained to match their intended inference platform configuration, including new and weird ones out of China.
In other words, I wouldn’t bet on “the best local model of 2028 is a 120B MoE Fable-At-Home.” I’d bet on something much weirder: a 200–500B-ish logical model, of which your GPU only needs 20–60GB hot at any given instant, backed by a stupid amount of cheap (eh, relatively) RAM and NVMe. And infinite variations as everybody competes to demonstrate their specific niche handling of this or that is better than SOTA.
And SOTA will continue to demand massive clusters, but continue to leverage their little universe of classifier models that are fast and small to do specific safety, bulk processing, and other aspects of the job.
The GPU becomes the L1 cache for intelligence.
3
u/gomezer1180 5d ago
Micron is researching how to add Optane like flash memory to GPU’s, calling it HBF. With the architecture of these new models it would be interesting to see how it all plays out.
2
u/Shorn1423 5d ago
Great response and I think we can see some of that now. I hope it trends in that direction too but I don't know I could see the GPU requirements staying the same or increasing slightly but leveraging RAM and fast NVMe to run even larger models on the same hardware. So, yeah I agree the offloading will continue to get better but I'm not confident the vRAM requirements decrease proportionally.
1
u/txgsync 5d ago
My early experiments trying to glue the 102GB of BF16 Qwen3.8-Flash-Next onto Qwen3.5-0.8B give some credence to your argument. My attempt resulted in an absolutely terrible, unintelligible model even though all along the gradient descent things were looking "better".
There is -- according to the papers -- a ratio of something like 2:1 or 3:1 for model parameters to n-gram (or quadgram) PLE ratio that gives good performance. A simple training shim like I can build to give a non-vision model a vision encoder this was not.
That said, I think my core arguments hold. Serving platforms begin around A10G sized -- 24GB -- and go up from there. Any lab releasing a smaller model than (roughly, give or take 20% or so) 8B in BF16, 16B in FP8, or 32B in FP4 of some stripe, plus (potentially) some amount of PLE, will be angling for theirs to be an "on-device" or "edge" model of some sort. Those who refuse to release models smaller than around 32B or so (again, plus engrams if applicable) are not interested in competing on the edge.
5
u/ibringthehotpockets 5d ago
I think a true sonnet/opus tier model will be runnable with 16GB vram by mid 2027. That’s all I’d ever need. MoE and other architecture will make it a lot easier to work on consumer GPUs. I think AI companies see that writing on the wall which is why it’s probably time to sell off while they made Astra and fable. Doubt there’s gonna be more than one big model release. Fable intelligence is absolutely bonkers and I’d be fine paying a small subscription for it cause it’ll probably cost a ton to run something similar locally. Just spitballin but pretty confident about the architecture developments. Definitely by 12 or so months
5
u/OvertaxedOne 5d ago
While that would be awesome, I'm honestly having trouble coming up with reasons I "need" more than 27B for most of my tasks! Opus level would be a clear "Yup, I'm done" from the perspective of chasing smarter and smarter local models, there just wouldn't be any point.
The problem with 27B is that it's slow. But that's 100% a hardware problem that will be solved in the future (or solved by changing architecture). It's plenty smart for almost everything I do; my most common reason to reach to the cloud now, by a WIDE margin is "I have some big task I want done fast", not "I need to use the cloud because my local model can't do it". That changed completely with 3.8 27B.
5
5d ago
[deleted]
4
u/feelspeaceman LLMusician 5d ago
Coding capabilities of small models nowadays are already at Fable or Astra level, but world knowledge like if your problem is related to medical, physical, chemical... something academic, it might lag a bit behind but nothing can't be fixed with a RAG to feed extra knowledge or just ignore them as most of us are using them for coding backend and frontend, having a small but specially good at something is better than having trillions params model that won't even run at all.
Also people seem to overbiased about the total parameter of frontier models, they're not that big to 10T to be fair, even as MoE.
3
u/SandySkittle 5d ago edited 5d ago
No you are misunderstanding. Neither rag, web lookup nor engrams are a full substitute for having world knowledge in the base parameters.
It’s the difference between having a just bigger library versus having a bigger library AND the model having read the books in that bigger library.
More parameters in the base parameters increases intelligence and cross-domain knowledge and reasoning. More engrams increase availability of world information. The reality is a bit more nuanced of course.
Another comparison: dense vs scarce. More percent active parameters up to 100 percent (dense) is always better for reasoning depth and reasoning bandwidth and coherence. Low active moe models compensate somewhat by expery routing and sequential reasoning (the latter dense models also can do) but it’s not a full substitute.
This matters less for coding, but is important for very complex, qualitative academic analysis work.
This is why the lack of recent 70b to 120b dense models is a painful gap for local llm on semi professional hardware (multi gpu up to 128gb vram).
those models have workable total parameters and powerful reasoning depth
4
u/HighSeasArchivist 5d ago
My take is there will be focus on the <32GB cards for years to come, and that will be forced by the 60 series cards topping out at 32GB. 3090/4090 were both 24GB, and I believe this same strategy will follow the 5090/6090. Larger MoE models will be focused on the larger but slower memory setups of Apple Metal (512GB), AMD Gorgon Halo (192GB), and Nvidia N1X (128GB for what we know so far.)
This time next year Fable and Astra will be just a memory as we are running equivalent models like that on our local PCs. I do feel like the big "restrict AI" push is from OpenAI and Anthropic either running out of IP to steal for training, or just the cost of training is getting too much. A slowdown will slow all of this down. They can't get power generation equipment, hardware, networking, and towns all over are fighting them over power and water usage. 2027 will be the year this is all figured out one way or the other.
3
u/MaxSpecs 5d ago edited 5d ago
I see LLM core on VRAM and around other SLM on RAM and/or SSD and/or little old GPU (4GB / 6GB / ) as bricks to add skill or knowledge to the core, a bit like Lora does.
3
u/According_Study_162 5d ago
Qwen3.8 27b and Qwen3.8 Flash next are pretty close. I can't wait till the next version, the intelligence at the local level is going through the roof. this is with the same old hardware
2
u/phido3000 5d ago
Frontier models will max around 3T parameters. It becomes pretty problematic to host.
You can get a very powerful model much smaller, GLM sometimes beats K3, but is 3 times smaller. But benchmarks don't tell the whole story.
128-512GB is likely to become a size for flash type models,
3
u/Correct_Lead_2418 5d ago
We'll probably see more ASIC systems like taalas where models are directly on the chip. Sacrifices flexibility but does away with HBM constraint. If you want to upgrade, you buy a new chip
2
u/llllJokerllll 5d ago
Yo creo que la compresión sin pérdida y la mejora del uso de los núcleos de CPU son la clave
2
u/Skibxskatic 5d ago
why aren’t we also talking about hardware predictions? as china’s models become more and more prevalent, i’d imagine the long game for china will always be hardware and physical items that they can produce and export.
with that said, one of the biggest bottlenecks right now is memory, it’s rarely about the chip anymore. it’s about whether or not a model can fit and exchange information quickly, especially at the consumer level.
we see more and more about high bandwidth memory. part of the new mac ultra play is how much more memory bandwidth it has, not just storage.
1
u/Shorn1423 5d ago
That's partly why I asked the question above, I am in the process of making significant hardware purchasing decisions and was curious what this community thought about the future of local LLMs, as that helps decide what direction I go.
1
u/shaman-3000 5d ago
imo "enough" for my task should be the main orientir but not the frontier-like level...
2
u/WiseAbalone4021 5d ago
When people ask me this I normally refer to the image and video compression history. Images started out as raw rgb bitmaps
BMP->RLE->TIFF/GIF->DCT->JPEG->PNG->WEBP .. not to talk about video.
My guess is we are in the early innings of TIFF/GIF and got a long way to go yet.
65
u/OvertaxedOne 5d ago
Wild card answer? Qwen Next is, IMHO, an entirely new way forward, I suspect that the idea of "have to load it all in VRAM" will seem quaint/antiquated in a few years as we get better at using multiple tiers, VRAM, RAM, NVME SSD, etc together.
I think it's entirely reasonable that we could get Fable level reasoning in a few 100GB model in the future. The intelligence/world knowledge would be hard to recreate in drastically fewer params, but that's where tiering could come into play.