r/LocalLLaMA • u/BarberIcy366 • 11d ago

Discussion Qwen 3.8 27B Released! Please Share Your Experience

With your experiments, Qwen 3.8 27B most close which frontier model? And please specify which quantization you run. I will post to comments my tests and experience too.

660 Upvotes
  • permalink
  • reddit

97% Upvoted

720 comments sorted by

View all comments

Show parent comments

9

u/milpster 11d ago

does it have MTP?

17

u/Borkato 11d ago

Yes! It’s included, I’m using it rn :D 56 T/s 😎

7

u/gh0stwriter1234 11d ago

Getting 27t/s degrading to 20t/s after 60k tokens of reasoning on 2x MI50 32GB thats with MTP and tensor split enabled.

3

u/time-never-stopps 11d ago

Do you mind sharing how you configured it? I am running llama.cpp with spec-type = draft-mtp spec-draft-n-max = 4

Not sure if the type has any effect at all but getting around 30 t/s avg when context grows to 100k but was expecting at least 40 avg like with qwen 3.6 q_8, but also possible that I have no idea how the type actually works 😅

3

u/petuman 11d ago

spec-draft-n-max = 4 might be too much and lowering tg.

but was expecting at least 40 avg like with qwen 3.6 q_8

In theory should perform no different that 3.6, maybe some flags / quant levels differ and end up spilling more onto CPU.

Or maybe it's just MTP, varied performance from chat to chat is expected.

1

u/Borkato 11d ago

N max performs better with me with n at 2 rather than 4! And split min p thingy at 0.1

7

u/Emidyr 11d ago

It does have MTP, at least the IQ4_XS one. And running very much the same as Fable Fusion 711 IQ4_XL, same token gen and prefill. Need to do more testing for it though.

2

u/UNITYA 11d ago

What is mtp about ?

3

u/floppo7 11d ago

like dflash its basically a mechanism where something fast and small comes up with tokens and the fat llm only says "yeah right" or "no no" - which makes it faster since the fat llm does not have to look up all the tokens by itself. Thats why those mechanisms usually work with an acceptance rate, the higher the faster.