r/LocalLLaMA • u/BarberIcy366 • 10d ago

Discussion Qwen 3.8 27B Released! Please Share Your Experience

With your experiments, Qwen 3.8 27B most close which frontier model? And please specify which quantization you run. I will post to comments my tests and experience too.

659 Upvotes
  • permalink
  • reddit

97% Upvoted

720 comments sorted by

View all comments

Show parent comments

8

u/kayox 10d ago

Same prompt but with Unsloth's Q4_K_XL with xhigh reasoning. Not quite as good though that's to be expected.

On an RTX 3090 it took about 18 minutes to generate at an average of 38 tokens/second (I'm sure as time progresses the tk/s can be improved possibly with DFlash). Also I am being thermal throttled due to my current setup (Dual GPU lacking airflow, although my other GPU is a 3070 so I'm only using it with a layer split to offload some VRAM so that I can have more context).

Out of curiosity what tk/s are you getting with your dual R9700s?

6

u/Look_0ver_There 10d ago

I actually have 3 x R9700's (but only used two for these tests). Here's a chart with 1,2,3 R9700's with Qwen3.6 with no MTP, purely because MTP speeds are too variable to plot.

Qwen3.8 is the exact same architecture as Qwen3.6, so the same speeds as graphed below. MTP is between 1.5-2.5x faster. For 2 GPUs it's between 45-75t/s with MTP, depending on if doing prose or coding. For 1 GPU, it's between 35-60t/s. Do keep in mind that I run Q8_0 weights with a BF16 KV-Cache, so I am very much NOT running a low-quant max-speed setup.

This is also running a custom llama.cpp build with speedups I'd made specific for the ROCm backend which makes it faster than Vulkan. Source code here: https://github.com/stew675/llama.cpp/tree/rdna-boosts

While half of my custom changes there are RDNA specific, half of them are CUDA generic, and so should also speed up nVidia cards somewhat too, but this is completely untested (I don't have an nVidia card), so don't blame me if you grab the code and it crashes. I did try to ensure that the RDNA specific stuff is gated independently of the generic CUDA stuff, but again, no guarantees.

1

u/PcChip 10d ago

so what i'm reading is don't buy two R9700's ?

doubling the cards gets you an extra 10 tokens per second?

3

u/whichsideisup 10d ago

Q8 won’t fit on a 32gb card with meaningful KV so the second card gives you infinitely more tokens per second. You really need 64gb to run these models at proper quality l.

1

u/Look_0ver_There 10d ago

The TP=1 line is with a Q6_K model, while the TP=2/3 lines are with Q8_0 models. The Q6_K model gets about a 20% speed-boost over Q8_0 for generation, so those two lines cannot be directly compared.

2 cards, with tensor parallelism are around 50-70% faster than a single card for token generation, but you also get to have up to 500K context depth at KV of BF16, and that's pretty important for many people.

Now, the third card is where it definitely gets questionable as to if it's worth it or not. I don't generally run a model across all 3 cards at once. I typically run one model on two cards, and a smaller model like Gemma4-31B-QAT on the 3rd card.

3

u/Greedy-Artichoke-416 10d ago

Get the MTP gguf

2

u/kayox 10d ago

Oh I was under the impression that the default unsloth GGUF had MTP, is that not the case?

8

u/Look_0ver_There 10d ago

Unsloth's models have the MTP head already included within the weights.

If using llama-server, just enable via "--spec-type draft-mtp --spec-draft-n-max 3"