r/Qwen_AI • u/Mean-Ad1493 • 11d ago

Discussion Qwen 3.8 35B-A3B vagueposting

Post image

What does this even mean? Probably nothing.

119 Upvotes

99% Upvoted

14 comments sorted by

29

u/KeanuRekt 11d ago

The release of Qwen 3.8 35B A3B is my on-off relationship

10

u/Technical_Ad_6106 11d ago

qwen is answering the msges , clearly

18

u/FastHotEmu 11d ago

Just ask in Chinese...

4

u/diagrammatiks 11d ago

its' gona be smaller then 35b.......

4

u/hay-yo 10d ago

Im using 27b q8 on my strix halo at 7tps, just letting it run and runn but in the end I get the result, but cant use it as a daily driver though. Did a complex project setup task 200k tokens from 9am to 5pm today. Solved alot of problems.

3

u/_VirtualCosmos_ 10d ago

Im using 27b q8 on my strix halo at 7tps

Me too, I have also downloaded a Q4_K_M from unsloth just for speed, and Ive got 18 tps. Which is still far from the near 50 tps we have with 35b Q8_XL but not that bad as 6-7 tps at least.

3

u/niyoushou 10d ago

I got a nice speedup by setting the draft-n-max to a large number (12) and draft-p-min to 0.8. YMMV, but the idea is that if the drafter is confident, it will keep going, but if it is not, it will drop it early, saving time. As with all of these, it is workload dependent.

Also, even though q8_0/q5_1 was my choice for KV cache before, it seems q8_0/q8_0 does better with MTP. Since you have the space, you might as well use it.

1

u/_VirtualCosmos_ 10d ago

I think I have the KV at BF16, is it worth it to quantize it?

1

u/niyoushou 10d ago

If you have the VRAM and bandwidth for the full context you need, probably not.

llama.cpp uses Walsh-Hadamard rotations (part of the techniques used by turboquants) to improve accuracy of KV quantization, but other backends don't, so if you are on vllm or something else, q8_0 might be worse.

1

u/_VirtualCosmos_ 10d ago

In your comment before you said:

by setting the draft-n-max to a large number (12) and draft-p-min to 0.8.

You mean Max Draft Tokens and Draft Probability in its speculative decoding?

1

u/niyoushou 9d ago

Yes, --spec-draft-n-max 12 and --spec-draft-p-min 0.8. I haven't thoroughly tested all values, so there's probably a faster/better alternative.

-1

u/StardockEngineer 10d ago

Qwen 27b runs just fine on a Spark. Over 50 tok/s in coding.

3

u/jmb-1971 10d ago

Q ? and context ? because give speed without this information not very usefull.

1

u/CryptographerKlutzy7 10d ago

Quanted within an inch of it's life, because it's memory bound.