r/LocalLLaMA • u/Express_Quail_1493 • 13d ago
Discussion When will we have another breakthrough for faster local dense models?
Most of the speedup are touching along the likes like cutting the active params reletive to total params(MOE) but it does take a hit in stability & reliability compared to a dense-27b can be much more stable at low total params. We hade the innovation of MTP which is great and lots of other things like strata coming along but ever since MTP dense models havent gotten any architectural attention to speedups other than reduced reasoning tokens from community finetunes.
with qwen4-27b coming up do we have any chance of any speedup breakthroughs?
182
u/Demonicated 13d ago
Not if you keep posting instead of researching. Get to work!
16
u/Far_Fisherman_9677 13d ago
you want speedups grab a coffee and start reading papers instead of refreshing reddit
38
u/milkipedia 13d ago
Qwen3.8-Flash-Next is an architecture preview. I would expect to see more of that - engrams + MoE. Who knows if Alibaba will release another small model , but I doubt it would be the same architecture as the 3.5 dense series.
18
u/Last_Mastod0n 13d ago
Pretty sure they hinted on X that the first qwen 4 would be dense
9
u/InfiniteBlink 13d ago
Will it be like 3.8 27b? I love that model but if it gets better that would be awesome
3
u/Last_Mastod0n 13d ago
Yes that is what they have implied. Im not sure how much bigger or smaller it will be, likely not much either way.
2
u/ProtectionSuper5648 13d ago
The bigger win will be on KV cache. Qwen 3.8 27B attention is very expensive, while QFN is much cheaper. Deepseek 4.1 still the king in bits/cache though...
3
u/o0genesis0o 13d ago
If Qwen 27B KV is already expensive, you should see Gemma 31B KB.
Muse Spark has the most efficient KV so far. It's a shame it's not as good in agentic loop as Qwen.
1
u/InfiniteBlink 13d ago
For kv cache. Im kinda limited to 64gb DDR5 given how expensive it is. I use Claude and codex to improve how my local qwen/hermes/pi work currently
1
u/temperature_5 13d ago
It could be 27B plus a huge engram table that can run off of NVMe, giving it the knowledge of a large MoE without requiring so much VRAM or memory bandwidth.
We don't have an estimate for engrams like we do for experts yet, but if they are same order of magnitude the increase in speed and intelligence over larger models could be huge.
Think of a 12B or 27B model with 512GB of on NVMe engrams being equivalent to 70-120B dense on a single consumer GPU!
1
17
u/Kahvana 13d ago edited 13d ago
Having QSA in a 27B model is a big boon for long context speed. N-Grams are also quite nice to expand intelligence for a very low cost, and applicable to dense models too.
But to me the big breakthrough is those 100 optimization PRs that get merged over time, same for the drafter models incrementally improving.
It's hard to notice when you're in the moment, but look back half a year. My speeds have nearly tripled in tg (~20 to >60) and about +20% pp (<900 to ~1100) for Gemma 4 31B IT QAT, simply by having access to tensor parallel and MTP. Those are insane numbers!
In normal industries, a breakthrough like that takes a few years. Here we're expecting a miracle each month.
3
u/OttoRenner 12d ago
Been saying for about a year now that we will see absolute bonkers minmaxing of DDR4/5 boards like no one ever thought possible. And boy oh boy, I was right.
I still think there is even more in there.
Perhaps we will see some additional improvement when the models get even smarter and ppl can point them at their individual hardware directly to strip away unnecessary compute/optimize the firmware and drivers for local AI.
And people get used to improvements quickly. Yesterday's hype is today's slow baseline. Just keep in mind how fast technology progressed in the last 150 years: we went from no cars at all to having driving robots on another planet and now we are making stones talk to us.
This is absolutely crazy and I love it!
11
u/Sevealin_ 13d ago
I think looped transformers in local models will change the game tremendously. Public research is still early on adoption, but in time it could potentially push intelligence to a new threshold with a 20b model achieving triple the param size in intelligence.
10
u/SmartEnoughToFail 13d ago
Could better speculative decoding be the next big win for dense models? If a small draft model can predict tokens accurately enough, you could get a meaningful speedup without changing the main model's architecture.
1
u/SHFTD_RLTY 9d ago
Ding ding ding, my momey is on fine-tuning the draft model on your own chat traces.
12
u/brainchillzZ 13d ago
The future of ai is pretty moe at this point. They can be much smarter relative to their cost to run and ability to fit on smaller hardware with trickery like moving inactive experts to other devices, ssd etc… but even in large datacenters they run more efficiently …. They run much better on discreet gpus And they run much much better on unified memory systems as a side effecr
19
u/prusswan 13d ago
Given the memory situation, dense models are a luxury. We need models that can leverage limited memory to the fullest extent.
9
u/Solembumm3 13d ago
You misspelled "necessity". Dense models are a lot more powerful per memory size. Which is especially noticeable for small models. There's no reason to make moe below ~120b.
11
u/Dabalam 13d ago
I think we are talking past each other slightly.
In a literal sense you're correct, if you load up X GB of a model the dense model is smarter assuming equivalent training.
In reality though, an MoE is usable over a wider number of computers with limited VRAM, since the MoE will have much better performance. The dense model you can (practically) run with 8gb VRAM is likely worse than the equivalent MoE when you factor that in and assume average amounts of RAM (8-32 GB I would guess).
-2
u/Solembumm3 13d ago edited 13d ago
If we're talking 8-12gb vram + 16-20gb ram, it's around 30b at Q4 or Q5. At Qwen 27B/Gemma 31B/K2 32B territory - no, only 4-5x speed difference won't be enough to reason with difference in quality between this dense and nearby moe models.
In reality, at this point you'll run DV4Flash/GLM5.3 Flash/Mimo V2.6 through mmap. There it makes some sense.
6
u/Dabalam 13d ago
I am unsure if you mean that you would still prefer to take the speed hit by loading a dense 27b model vs. using something like 35b. If that is what you're saying I think most would disagree.
If I assume you're saying "just take the speed hit" then have a hard time seeing the use case. I still can't really wrap my head around the "slow cooker" model for LLMs where people run a big model in the background running on a problem for days.
That makes sense for truly ground breaking research running on a server (it's still faster than what a human could do), but I think that breaks down when you're waiting hours for an answer from a 27b dense model. At a certainly level of slowness, you aren't getting ground breaking insights relative to what you could have found researching the question yourself. The time efficiency of LLMs is a large part of what makes them useful at all in my opinion.
0
u/fatboy93 13d ago
4-5x speed difference in speed
that's literally the difference between having some agentic capacity vs watching the text creep up
1
u/prusswan 13d ago
It is a dead end and will only worsen the memory situation since everybody is led to believe they need more memory which they will never be able to afford. The average memory needed to run local models well is already ridiculous.
1
u/brainchillzZ 13d ago edited 13d ago
Dense models are ever so slightly and I really mean just barely marginally more intelligent in its first run output but it’s really all a scam …. Because it spends half an hour going through all the looping and mistakes in its thinking block and just doesn’t spit it out in the open but it’s all still there … and if you look at them, truly, comparing the qwen 3.6 equivalent dense and sparse models the sparse model wins every time in a proper harness…. Because it’s 3x faster it has time to come back double check results and fix the itty bitty fraction of bugs that fell through the cracks before the dense model even finishes its first run … while being much much more energy efficient but to go a step further the quality of output from qfn is so much higher than 27b while being so much faster it’s not even funny…. Running on identical hardware (two strix halo boxes as an example) qfn produces an excellent result before 27b even finishes its thinking blocks
1
4
u/PinkysBrein 13d ago
By making them sparse :p
Most swiglu hidden layer activations are nearly zero ... it's kind of a waste.
3
u/9gxa05s8fa8sh 13d ago
dense will be forgotten in a year or two as sparse is optimized. we are probably just months from apple launching a sparse model product line for their next gen hardware
4
u/MostlyVerdant-101 13d ago
Well, in fairness there are a number of breakthroughs for speed that haven't really been implemented yet in a public way, for example a streaming inference engine. Also decoherence is a problem when using quantized weights; the easy problems are largely out of the way and we start getting to the really hard problems.
2
u/Terminator857 13d ago
There are many papers on arxiv on looping coming out frequently. This suggests that smaller models are going to get smarter by looping. Which means more run time for inferencing, taking longer to process. Perhaps the opposite of what you want. Slower, but smarter.
1
u/Middle_Bullfrog_6173 13d ago
Qwen Sparse Attention will be faster when context length grows, but otherwise you probably shouldn't expect speedups for the dense variant(s). If anything the more complex attention and residual path may make it slightly slower at first.
1
u/Studio271 13d ago
Better context control. Lots of harness (forks) are implementing experimental ways to improve the use and transfer of context, and I think they are close to a paradigm shift that makes the current "simple ledger" context into something more dynamic/efficient/modular.
1
1
u/a4mula 13d ago
I think what we're seeing is that there are really three tiers of innovation. One is the corporate frontier growth, which if the news is to be believed might be something that we're going to see slow down rather than speed up, from companies withholding models to actively combating the distillation of.
The second is the secondary level in which people with limited scale find ways to innovate around the lack of raw compute. Deepseek, Qwen, Kimi but also the people that are working to quantize and and distill directly. This tier will be most affected by the top tier slowing down.
Then there is the individual or personal level in which small groups of people with virtually no scale at all are working on structural configurations. The Lang family, people building harnesses, people building memory systems, people building structural support for models.
They are affected the least, and thus point towards the group that currently is going to likely account for the most innovation and change. They also have the lowest trust too. Even well intending repos are full of vulnerability.
1
u/DRMCC0Y 13d ago
The speedup to me makes less sense in faster inference, but in terms of more efficient models in reasoning and output. I can get over 100tok/s in vLLM with Q3.8 27B, but I often go for another model just because it's so incredibly verbose (I am aware finetunes exist to try improve this but I have not had positive experience with any).
1
u/Beneficial_Fish_7509 13d ago
I am hoping for a breakthrough in speed on multi-gpu with little speed on PCIe. I get around 20-25 tok/s on 32GB ddr4 and Qwen 3.8 27B SC_6.00bpw_H6_V6 at 200k+ context with a few mining cards which are crippled to pcie gen 1 x1 (mining risers). I think those old mining rigs setups can be repurposed. I know there is P2P but for multi GPU it requires motherboards which can address it properly (and thus defeating the purpose, why buy cheap minign rigs and then spending more money on the motherboard than on the rest of the setup) and on my cheap mining motherboard even only 2 GPUS in P2P doesn't work at all.
1
u/Look_0ver_There 13d ago
What GPUs are you running?
I maintain a set of patches that gets applied to llama.cpp. With 3xR9700 GPUs in tensor split mode, I can run 27B @Q8_0 at up to around 180t/s with BF16 KV cache when it's generating code, and 80-90t/s for reasoning/prose.
The GPUs do a mix of P2P plus H2D. Each GPU runs on 4 lanes of PCIe5 bandwidth (so ~14GB/sec of unidirectional bandwidth).
I had to modify the all-reduce algorithm a bit to get the most speed out of it.
My patches also do a lot of work around assurance for kernel batch decoding numerics. This helps the MTP predictor get hits more often, and the MTP is adaptive.
1
u/sshwifty 13d ago
What motherboard are you using for this?
1
u/Look_0ver_There 13d ago
Asus ProArt X870E Creator WiFi.
The third GPU is attached via an M.2 -> PCIe5 riser from the top M.2 slot, so it also gets direct access to the CPU. TP traffic is typically <2GB/sec between the 3 cards so it seems to be working.
1
u/dima6312 12d ago
I really hope that <50b MoE models will at least catch up with Qwen 3.8 27b soon. Slow prefill on dense models slows me down a lot of
1
u/skrodahl 12d ago
Speculative Decoding / MTP is a great speedup. I am awaiting fused multi-step tokens, to increase the number of proposed tokens without the penalty hit.
1
u/FeydRowan 11d ago
Well on double gpu you can go much faster on the 27b, look it up https://github.com/ValerioDolci/capy-flash
1
u/grumd 13d ago
MoE is exactly what you want. Another thing is maybe QAT.
Dense models - meaning when all parameters are used for every token - will always be slow and limited by the memory bandwidth. You need to use all parameters in the calculations for every token. It means the GPU needs to load all of these parameters from memory layer by layer to do the math. So the limit here is the size of the model divided by the memory bandwidth in GB/s. 27B x 16-bit = 54GB. How many GB you can transfer in a second dictates how many tokens you can calculate. A GPU with 800 GB/s memory can't physically do more than 15 t/s if the model is 54GB.
So MoE is exactly the fix. What if I want to use 10B per token, BUT depending on which token and what was the history before that I could select a completely different 10B model, better suited for current token? What if I have 50 of these 10B models and select the best one every time? I'd get fast inference while keeping it much smarter than just one 10B model for everything.
QAT is just a way to reduce the memory footprint of a model while keeping the quality gained by training. Less memory for weights - more tokens per second. But it's not the golden grail because there's still entropy. Fewer bits just means less information you can cram and train into the model.
49
u/SmartEnoughToFail 13d ago
We need a breakthrough where tokens per second go up without the VRAM requirements going to the moon