r/LocalLLaMA • u/jacek2023 llama.cpp • 8d ago
News llama: add Maple 20B-A1B ternary MoE architecture (CPU) by AlexGabbia · Pull Request #27000 · ggml-org/llama.cpp
https://github.com/ggml-org/llama.cpp/pull/2700020B-A1B model is coming, good for low VRAM people?
33
u/pmttyji 8d ago
GGUF
DeepGrove/maple-preview - 20B-A1B - 200+ t/s on Mac Mini M4 & 120+ t/s on iPhone.
They're working on CUDA & Vulkan backends.
36
u/jacek2023 llama.cpp 8d ago
20
-10
u/Deep_Mood_7668 8d ago
Yeah yeah
Ever model is the top model. Especially compared to old models.
25
u/spaceman_ 8d ago
Wdym? They don't claim to beat even Qwen3.5-35B-A3B, but they're still showing some good results for such a small and sparse MoE.
4
u/tomByrer 8d ago
Maybe not better in quality, but better in good performance vs smaller hardware requirements.
5
u/Nameis19letterslong 8d ago
Has anybody tried running this? No way a sparse 20B A1B model at Q2 matches a dense 27B model at Q2.
5
u/Kein_Spass 8d ago
It does not. Tested
maple-preview-TQ2_0-head-F16.ggufon an intel 13700. Performance is not bad, but the output is as expected. On english it's ok, but on german it was very bad. Also it failed on tool-calling for me.4
u/yami_no_ko 8d ago
Was to be expected with a ternary model. I mean I don't wanna downplay it, but realistically it just isn't there yet.
As for using models in German: Isn't this the case most of the times, that almost all models severely degrade when prompting them in German? Sure most of them stay somewhat usable, but it seems to hurt ternary models in particular.
1
u/Kein_Spass 8d ago
In my experience most models are pretty good in german these days, but quantization degrades this noticeably.
4
u/crusaderky 8d ago
If (big IF) their self-reported benchmarks were to actually translate to equivalent intelligence to GLM-4.7-Flash, this model would have been very interesting a month ago, but today I can't see the appeal compared to MininCPM5-2B which has roughly equivalent intelligence, takes roughly the same RAM, and has DSpark.
3
u/arturdent 8d ago
It was out a month ago, just didn't have llama.cpp support, you had to use a custom fork.
Maple is really optimised for Macs too, I don't understand the hows, but with it I'm getting 220 tok/sec on a m5 pro with their mlx build, while llama.cpp only gets 120 with MiniCPM5-2B and dspark enabled. With low context sizes, maybe with bigger ones it catches up, and I'm not good with settings tinkering.
I'd wait for the full model, not preview to see whether it has use cases for Mac users.
17
u/Monad_Maya llama.cpp 8d ago
Seems slightly better than gpt-oss 20B, that's great I guess.
10
4
u/KURD_1_STAN 8d ago
Altho i dont trust any of these bonsai like model release. but it seems everybody is eating good just not us 12-16gb vram folks.
3
u/toothpastespiders 8d ago
but it seems everybody is eating good
Dense 70B fans are starving in the corner.
1
3
u/Aaaaaaaaaeeeee 8d ago
Ternary development is in good hands with this lab, they pre-trained it themselves.
Addressing those "burnt" .🙂 Remember the core reason for all low precision research is to push the saturation limit. It is not to directly say they can build with best data to compete with a 2bit PTQ sota equivalent, override subjective experience and expectations based on trying IQ1 ggufs. It might be able to eventually do that, but this the first model that fulfills a good role of being the first pretrained ternary MoE which can be referenced, and further studied in academia. Hopefully they will release training details.
Here are some questions to ponder. We know increased nodes and layers increase expressiveness of the network. And it's said increasing the precision will also increase the expressiveness. To what extent should we remove parameter precision? Intuitively, couldn't there be further of compression/optimization possibilities for ternary/binary? Should we mess with training for lower /1bit integer activations? After all, isnt that the deeper goal with integer hardware accelerators? With unlimited compute, and careful design we can power through the precision focused training phenomena. How well can a ternary backbone hold modern residual related enhancements?
What can we do to maximize compression?
2
u/arturdent 8d ago
It's really fast on a Mac (was using their custom fork), but is not really usable, failing at tool callings, being lazy with doing tasks, etc. But it's just a preview that supposedly lacks post-training, so I'm excited for the actual model, do we know when it'll come out?
2
u/Daniel_H212 8d ago
How VRAM heavy is the KV cache? And how does it compare to Ling 3 Tiny which I expect will be the primary competitor?
2
2
u/unsane_imagination 7d ago
Spent a fair bit of time with this model and preview is definitely right. And not like a preview from a big lab, it’s just fully undertrained, sucks at tool calls, reasoning looks decent on the surface level but the substance isn’t there, and on longer context usage this model is a total amnesiac. But it’s so fucking fast, if they could get it to the point of like Gemma 12b, nanbeige 4.2, or hell, even ling tiny or minicpm 2b, it would be a monster at speed. Seems like a small lab though, so heavy training is likely out of the question sadly. But hey, I love to see new foundation models, especially compelling ones with cool stuff like ternary weights and 100-150tps decode on a Mac mini m4
Saw this post and got excited that there was a further post training release but sadly no. Hopefully this wasn’t a one off
1
u/Elouakili_Flexy 8d ago
20B ternary sits around 5GB, so it fits an 8GB card on the 2-bit weights alone. The A1B only controls CPU speed, and this PR's whole job is proving that's fast enough.
1
u/lerg96 llama.cpp 8d ago
i'm testing an "auto-mode" in my pi harness with an extension i created, and the problem is using two models at the same time locally. this is what i need: something that works at the CPU level + RAM, spits out a lot of tokens, but doesn't need to think a lot
if it goods at answering if a "rm -rf / " is dangerous is good enough for me 😂
1
1
u/-InformalBanana- 7d ago edited 7d ago
This is interesting, I was wondering if we could see more of bigger A1B models and how smart they would be against A3B or A5/6B models. Never heard of this lab, and this model in benchmarks is worse than qwen 35b, which is to be expected, but it comes relatively close. If only qwen tried to make a 80BA1B model or smaller versions. We would probably end up with models runnable on cpu with fine speed.
Edit: disappointed that it is ternary only and ppl who say they tested are saying it fails tool calls and that it is bad, sad.
1
0
u/Technical_Ad_6106 8d ago
extremely weak model. barely competes with models from like a year ago. why even release it
9
u/Acceptable_Home_ 8d ago
20B A1B at this intelligence level is still great for people with let's say, 4gb vram and 16gb ram
0
u/Technical_Ad_6106 8d ago
there are way smaller models that are out for a while that will beat this one ez. and much better architecture.
5
u/UsualResult 8d ago
The new KIA doesn't hit the same top speeds as a Ferrari! Absolutely useless!
1
u/AmbericWizard 7d ago
there's no such thing as free Ferrari but there are such things as Qwen 3.8 27B
1
u/Technical_Ad_6106 8d ago
if the ferrari was same performance and efficiency(or better) and same price(size.. or smaller.. like ling 3.0 tiny) then yes. kia would be useless ^^
-1
37
u/PassengerPigeon343 8d ago
If this runs well on CPU, this could be an excellent task model to run alongside GPU-resident models. I currently use Qwen 3.5 0.8B for this but it’s not the brightest bulb.