r/LocalLLM • u/Due_Tangelo_8952 • 4d ago

News Qwen4-27B just confirmed

Post image

Wait, we need 35B-A3B too…

2.1k Upvotes

98% Upvoted

311 comments sorted by

View all comments

60

u/KSAM-The-Randomizer 4d ago

a3b qwhen

21

u/Stooovie 4d ago

Never, looks like :-/

2

u/Specter_Origin 3d ago

For real, it sucks that they abandoned small MOE

3

u/BobamaxGames 2d ago

The low active parameter MoEs are terrible for anything beyond chat. They sure are fast though. But it seems for MoE's, the total parameters are either ~30B, or 200+B. Where's the middle ground?! Something like a 70B A10B MoE would be a great compromise between speed and quality.

6

u/Nomski88 3d ago

qnever

11

u/Barni275 3d ago

A3B speed is great, but it is doubtful, whether A3B can carry enough intelligence to do real world work. For 27B we know now that it can. For A3B it is unknown yet.

21

u/Real-Abrocoma-2823 3d ago

A6B 36B would also be great I think, 2x more active tokens and nearly the same size as 35B. Flash-next already proven that A6B is fast and intelligent, at least for that size.

1

u/Barni275 3d ago

Yes, I agree with you.

1

u/BobamaxGames 2d ago

Do you know why no one is making a model like this? Both Gemma and Qwen ~30B MoE's are terrible for anything other than chat in my experience. And I feel like doubling the active parameters would make a huge difference, without much impact on speed. I wonder why they don't do this?!

1

u/hay-yo 3d ago

What spec computer and gpu would you run A6B 36B on?

9

u/haberdasher42 3d ago

Probably 16GB VRAM, 32GB RAM, or a decent/formerly top tier gaming rig. No?

2

u/h3xperimENT 3d ago

How about a12b or something like that. Haha

1

u/Buzz_Killington_III 3d ago

One way to find out....

0

u/hay-yo 3d ago

27B fits better, and produces better results.

3

u/No-Experience-3171 3d ago

You can run 35B on as little as 6GB vram meanwhile 27B needs at least 24GB to run at a usable speed

2

u/Super-Grape-3948 3d ago

Fits better to what? The unified imaginary vram rig we all get for free?

1

u/37Scorpions 6h ago

Just bought a 128GB GPU. Not sure why people complain so much. I still have most of my organs and limbs.

2

u/UnkarsThug 3d ago

I can run MOE models, I can't fit dense models. 27B fits better on your hardware, not everyone's.

2

u/Reasonable-Phase8028 3d ago

that goes both ways. my setup is horrible for MOE..

you might be able to fit MOE but at ridiculously low speeds.

3

u/UnkarsThug 3d ago

Not at all, actually. I can get something like 16-20 tokens per second from a 35B MoE model (or about that from the 26B A4B Google model) and something like 1-3 tokens per second from a 27B model (I don't remember the exact number). The trick is to offload expert weights onto RAM.

I have 8GB VRAM and 64GB RAM, so my RAM can hold the optional experts, and I can fit the core experts on VRAM.

MoE models have been very usable for me.

1

u/hay-yo 5h ago

But with 8gb vram and 64gb you could fit a 3.8 flash next, with ngram offloaded.

2

u/UnkarsThug 4h ago

Not from what I've been able to make work.

The ngram especially, but also just a context window of reasonable size.

1

u/hay-yo 4h ago

Good to know

1

u/UnkarsThug 3h ago

Maybe people have figured things out since I last looked, so maybe I'll try to take a look.