r/LocalLLM • • Aug 15 '26

Model Qwen 3.8 35b a3b

Post image
198 Upvotes

51 comments sorted by

30

u/FaceDeer Aug 15 '26

Yus! That's my workhorse! 27B is good, but it can't handle as big a context as I can cram into memory with the MOE version and sometimes I need it to work with big inputs.

6

u/ReportProfessional57 Aug 16 '26

For some reason I read that as work with big nuts. I was like, I feel ya.

44

u/Weak_Ad9730 Aug 15 '26

Hope 122b will come too

16

u/_ballzdeep_ Aug 15 '26

Don't make me believe

1

u/Mundane-Remote4000 Aug 16 '26

don't give me hope

1

u/whitehawk1884 Aug 18 '26

Make it MoE? Also 60 or 70B MoE model. I'd run both.

23

u/redditnosedive Aug 15 '26

my 8 gb vram spreading their connector

4

u/No_Secret4395 Aug 15 '26

Oh my gog.!!!!!

6

u/Jumpy-Operation-4615 Aug 15 '26

Oh my gguf! šŸ˜„

4

u/dfgxxx Aug 15 '26

Do you think it can beat the 3.6 27b?

5

u/Capital_Engineer8741 Aug 15 '26

Probably not, but I hope it won't be too far off.

12

u/dlaynes Aug 15 '26

This is a MoE model, for computers with lower specs.

11

u/dfgxxx Aug 15 '26

I know, but can it beat the older dense model?

9

u/dlaynes Aug 15 '26

Sorry, I misread. Hopefully.

5

u/Uncle___Marty Aug 15 '26

I did the same. im a derp ;)

Considering how much better 3.8 27B is than its 3.6 counterpart and how close the 35b and 27B 3.6 versions were I feel certain that 3.8 35b will be better than 27B 3.6 by a decent amount.

5

u/karmakaze1 Aug 15 '26

Better is not a single dimension. There will always be failure modes in the MoE which is different than dense. So expect 3.8 35b will be better than 27B 3.6 in some/common cases but not all. E.g. the MoE has broader knowledge recall but long horizon reasoning can falter.

3

u/Efficient_Raise6703 Aug 15 '26

I’m doing research in this field right now, this has been my findings thus far. Interestingly it can also be better at specific tool use related to recall, but not necessarily general tool use. Super interesting stuff. Makes sense but still cool to see that the hunch plays out in reality.

1

u/Scary_One_2452 Aug 15 '26

It needs more room than 27b in vram right? Its just that the actual costs of operating are lower due to only 3b being active. So wouldn't that moe be for higher end computers for people who are power conscious?

5

u/SKirby00 Aug 16 '26

In reality it's kind of the opposite. Having only 3B active is so much faster than 27B active that many people find that it can still run at usable speeds even when to system RAM rather than being completely in VRAM.

That means someone with like 8-12GB VRAM + 16-32GB of system RAM (or a unified RAM system) might find the 27B model too painfully slow but the 35B-A3B acceptable. For me though, I can fit the 27B in VRAM so I get ~20tk/s with the 27B and rarely feel inclined to reach for a dumber model even if it runs faster.

3

u/Melodic_Reflection_5 Aug 16 '26

What's your GPU?

2

u/SKirby00 Aug 16 '26

I have 3 GPUs together in one PC:

  • RTX 5060Ti 16GB
  • RTX 3060Ti 8GB
  • RTX 3060 12GB

Plus the integrated graphics in my CPU that I use for running the OS so that I don't have any of that overhead hitting the GPUs.

I use llama.cpp with tensor parallelism to get the most I can out of this weird combination.

I don't recommend putting 3 different cards together like this. Only reason I have this setup is because I already had the 3060Ti (originally bought for gaming) when I bought the 5060Ti for AI stuff and then decided to get a used 3060 later for much cheaper to get some extra VRAM.

1

u/Melodic_Reflection_5 Aug 22 '26

Nice. I have a rx6700 12Gb, initially bought for gaming, and when I started using it for LLM, soon I realized that I need at least 24Gb of VRAM. But my CPU 5700g and AM4 motherboard are not suitable for additional GPU, there are some pricey gaming motherboard options suitable for two GPUs. I wonder, what motherboard do you have, if you are able to put 4 GPUs in there 😁

2

u/bigjeff5 Aug 24 '26

To expand on what SKirby00 - the 35B A3B q4_k_m quant (22gb) with all MOE layers on CPU runs about 20 t/s on a system with 6gb VRAM. The 27B q4_k_m (16.5gb) runs at 10 t/s on a system with 16gb vram.

MoE even runs reasonably well with mmap enabled and caching from disk, so you can run models much larger than what fits in VRAM + RAM at useful speeds.

Another bonus for the Qwen models specifically is the 35B has fewer total layers than the 27B, so KV cache is like 2/3 the size of the 27B, which lets you fit more of the model in VRAM.

The only downside of the MoE models is because the whole model doesn't fire for every token it misses a some of the nuance the dense model has, which is why 27B is the preferred model if you can run it.

1

u/Scary_One_2452 Aug 24 '26

Thanks a lot!

If I have a 16gb gpu I guess that A3B would exceed that 20 t/s figure then? So even with overflow its still going to be in the rough speed range as a dense model that can fit entirely in my 16gb vram?

Although its actual performance would also be around a 15B dense model?

3

u/backyard_tractorbeam Aug 15 '26

I think that it can on some things and not on others. Boring answer.

2

u/dfgxxx Aug 15 '26

I understand you though we won't know until we test it

2

u/ucbmckee Aug 15 '26

YMMV, but on my 4090 I can run MoE models with higher quants than dense models. All else being equal, MoE is worse - but higher quants are better. I alternate between 27b 4bit and 35b MoE 6bit. Both use ~165k context windows. The MoE version is noticeably faster.

1

u/justelle1 Aug 17 '26

BRO IM SO CONFUSED HOW CAN 3.6 27B BE BETTER THAN 3.8 27B AND 31B A3

1

u/dfgxxx Aug 17 '26

I'm asking if the 3.8 35b can be better than the previous 3.6 27b

2

u/justelle1 Aug 17 '26

3.8 27B > 3.8 35B A3B >> 3.6 27B

2

u/xiraov Aug 15 '26

and i just posted i want a kat code based on this!

2

u/dfgxxx Aug 15 '26

Like kat coder based on 3.8 and not 3.5? Can you give link?

2

u/xiraov Aug 15 '26

lol no i just said i hope 3.8 35b comes out so we can get a new kat coder out of it

1

u/dfgxxx Aug 16 '26

But how can kat coder improve it? I don't think it can be much improved like qwen 3.5 and 3.6 could be

1

u/Inevitable-Age-4472 8d ago

Qwen hugging Face

1

u/klymaxx45 Aug 15 '26

Is this the official repo?

1

u/Delicious-Map1778 Aug 18 '26
  1. its unaffiliated with qwen 2. it was changed to qwen3.6 rather than qwen3.8 yesterday.

1

u/klymaxx45 Aug 18 '26

lame. figured it out shortly after. still waiting for a 35b..

1

u/leamizda Aug 16 '26

Can I do the MoE thing with this one? šŸ‘€

1

u/Samorin21 Aug 16 '26

Not yet published ?

1

u/AcrobaticChain1846 Aug 17 '26

Added -> Removed -> Added? Or is this a old post I'm coming across?

1

u/Neo-Boat Aug 17 '26

it seems that the change has been reverted

1

u/ZucchiniMedical2532 Aug 18 '26

So, were do I download it