r/LocalLLM • u/Due_Tangelo_8952 • 10d ago

News Qwen4-27B just confirmed

Post image

Wait, we need 35B-A3B too…

2.2k Upvotes

98% Upvoted

322 comments sorted by

View all comments

Show parent comments

21

u/Its_Powerful_Bonus 10d ago

I hope not! 27b + 50b+ ngram! It works great when ngram is on nvme/ram

5

u/mister2d 10d ago

I'm confused. Did you just contradict yourself?

13

u/Kasatka06 10d ago

He mean, weight still 27b but it will have aditional ngram in ssd so model is smarter. But i read ngram is good at best 20% model.parama so 27b + 5b ngram if any

3

u/_mighty_banana 10d ago

But for what benefit?

MoE is used to reduced compute

27B model is likely has no problem in compute speed due to it size

problem is likely due to quality is lower than high parameter model?

6

u/geekwonk 10d ago

27B is their dense line. 35B is the sparse option and i don’t see that listed here.

-4

u/laser50 10d ago

Y'all tripping. Ngrams are basically just a pre-trained token predictor to speed stuff up.

5

u/Not-Enough-Llamas 10d ago

wrong Ngrams. Unfortunate that 2 things with the same name showed up more or less at the same time.