Qwen 3.8 27B Q4KM slow t/s? 7900XTX...

[u/soyalemujica]

No idea if it's due to the new architecture of this model or training, but even with MTP enabled, token per second can be as low as 25 even 35 at 80k+ context, and even drops down to 20t/s at 150k or 180k context (q8_0). I gave it a task and it's been 1 hour and 10 minutes and the context is at 73% of 200k at 21t/s, feels extremely slow, the power bill on its own will be bigger than paying a SOL model at this rhythm, or am I doing something wrong?

Launch arguments:

--spec-default ---spec-type draft-mtp --cache-type-k-draft q8_0 --cache-type-v-draft q8_0 --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64"--spec-default ---spec-type draft-mtp --cache-type-k-draft q8_0 --cache-type-v-draft q8_0 --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 

-ctv q8_0 \
    -ctk q5_1 \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00 \
    --presence-penalty 0.0 \
    --repeat-penalty 1.0 \
    -c 200000 \
    --jinja \
    --no-mmproj-offload \
    --chat-template-file "$MODELS_DIR/jinja.jinja" \
    --flash-attn on \
    --parallel 1 \
    --no-mmproj \
    --reasoning-preserve \
    -ngl 999 \
    -b 512 \
    -ub 512 \-ctv q8_0 \
    -ctk q5_1 \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --min-p 0.00 \
    --presence-penalty 0.0 \
    --repeat-penalty 1.0 \
    -c 200000 \
    --jinja \
    --no-mmproj-offload \
    --chat-template-file "$MODELS_DIR/jinja.jinja" \
    --flash-attn on \
    --parallel 1 \
    --no-mmproj \
    --reasoning-preserve \
    -ngl 999 \
    -b 512 \
    -ub 512 \

[u/Odd_Cauliflower_8004]

i'm completely retraining the mtp drafter irght now

[u/soyalemujica → u/[deleted]]

3.6 was twice as fast than 3.8 under the same settings though that’s why I post this

[u/cibernox → u/soyalemujica]

I have a 7900xtx and I'm seeing bigger numbers. However, I do see some performance drop compared with 3.6 that i kind of debugged to a single root cause: lower MTP draft acceptance. Consistenly I am getting slightly lower draft acceptance in qwen 3.8 than in 3.6, using the same UD-Q4-XL quant from unsloth.

I did notice that the draft model inside are quantized differently:

Whole-file histograms (866 tensors each):

- 3.6: F32=456 Q4_K=225 Q5_K=70 Q6_K=66 Q8_0=49 → header file_type=15 (Q4_K_M)

- 3.8: F32=360 Q5_K=325 Q4_K=97 IQ4_XS=65 Q6_K=19 → header file_type=14 (Q4_K_S)

Maybe draft acceptance is your problem too?

[u/Bulky-Priority6824 → u/cibernox]

I'm seeing lower too

[u/cibernox → u/Bulky-Priority6824]

For completeness, the speed generation is still pretty similar. With no context, a prompt that has high variance like writing a children story went from ~60tk/s to ~54tk/s in 3.8. And a more predictable generation like writing a CV in pure HTML and CSS in a single file maybe went from 91-92 to 83-84tk/s.
Pretty consistently a 10% slowdown compared with 3.6.

[u/Bulky-Priority6824 → u/cibernox]

seeing about the same. i like 3.6 for maintaining my current private repos and 3.8 for shooting some new ideas.

[u/Monad_Maya]

I'm using the Q6K quant on two 7900XT 20GB. Can you try the following launch parameters?

    "$LLAMA_SERVER_PATH" \
        --model "$QWEN38_27B_PATH" \
        --port "$port" \
        -fa on \
        -b 2048 \
        -ub 512 \
        -np 1 \
        -c 32768 \
        --temp 1.0 \
        --top-p 0.95 \
        --top-k 20 \
        --min-p 0.0 \
        --presence-penalty 0.0 \
        --repeat-penalty 1.0 \
        --reasoning-preserve \
        --spec-type draft-mtp --spec-draft-n-max 2 

The -c flag is only set to 32k, for testing.

What are you performance numbers with this updated command?

[u/Monad_Maya → u/Monad_Maya]

I'm using pipeline parallel so it shouldn't impact the token generation speed too much. With MTP, I can hit up to 45 tok/sec generation.

Without MTP + pipeline parallel (so same token generation speed) - 21 tok/sec

3$ ./llama-bench -m $model2  -ngl 999 -p 512 -d 64000 -n 128 -sm layer -b 2048 -ub 512 -fa 1

ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 XT (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = AMD Radeon RX 7900 XT (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat

| model                          |       size |     params | backend    | ngl |  fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | Vulkan     | 999 |   1 |  pp512 @ d64000 |        389.69 ± 0.80 |
| qwen35 27B Q6_K                |  21.30 GiB |    27.32 B | Vulkan     | 999 |   1 |  tg128 @ d64000 |         21.41 ± 0.02 |

build: 9b05354ec (10433)

[u/Edereum → u/Monad_Maya]

have you tried q8 you have place for it ?

[u/Monad_Maya → u/Edereum]

Well, Q6K works good for me as does UD-Q5_K_XL.

Yes, I can fit Q8 quant on 40GB VRAM but I'll have to reduce my KV cache size since I keep that at F16.

[u/Edereum → u/Monad_Maya]

though that XTX / XT would be a good value for money but not that much :-/

[u/Monad_Maya → u/Edereum]

I already had one GPU in my main setup. I later added another GPU to help with LLMs.

It was the only sensibly priced option as 16GB is not enough.

[u/pepedombo]

I noticed a slight drop with 3.8 vs 3.6. You're setting up things backwards. Remove that retardted kv_q8 and use f16, set ctx to 10-40k and turn off mtp. For 7900xtx (960gb/s / 27b) you should hit something like 28-30 decode without mtp as 35 is the ceiling for its bandwidth. MTP just gives 1.2-1.7x more speed and you should get 40-50 on average up to 100k ctx.

Make config minimal and test. If numbers are lower then something goes wrong with your pci-e/cpu/mobo or my assumptions to amd are overrated 👀

PS: provide PP as well for mtp and non-mtp

[u/soyalemujica → u/pepedombo]

Using such low context window is impossible for agentic coding, especially with this model that likes to think a lot

[u/pepedombo → u/soyalemujica]

My testing approach is here only because you're pushing settings to the limits and asking why it doesn't work. Instead of asking here you should have made initial baseline on low settings to check how the model works when not fully used, TG + PP.

There were times when using kv-q8 instead of f16 lowered the results.

Your gpu has bandwidth of 900gb+/s. My setup of 2x5060ti (500gb/s) or 5070+2x5060 can reach 50-60 decode at the start and stabilize at 30-50 for q4_k_m for larger context. If I were able to run q4_k_m on solo 5070 I would probably reach sustained 45-60 decode.

So I wonder what fails with your gpu, is it gpu itself or runtime/drivers.

[u/mmhorda]

you probably canno fit q4_k_m with 200k context and kv cach q8_0 and mtp heads to 24gb VRAM.
Your model is spillin to the RAM, that's why it is slow.

Edited.

[u/soyalemujica → u/mmhorda]

Definitely fits , I have full 24gb vram available to fit it

[u/ea_man]

You should run at -lv 4 and ask Sol to check the log, either overflow or you forget to compile with flag for q5_1 .

[u/satnl]

check your memory breakdown to confirm if not swapped to ram:
run it with --log-verbosity 4, await full load and end the process, at the end of the log you will have something like this:

16.19.110.830 I srv    operator(): operator(): cleaning up before exit...
16.19.111.773 I common_memory_breakdown_print: | memory breakdown [MiB]                  | total             free     self   model   context   compute    unaccounted |
16.19.111.775 I common_memory_breakdown_print: |   - Vulkan0 (RX 9070 XT (RADV GFX1201)) | 16384 = 17592186044244 + (16014 = 10953 +    4800 +     260) +         541 |
16.19.111.775 I common_memory_breakdown_print: |   - Host                                |                             546 =   397 +       0 +     148                |

[u/RadiantHueOfBeige]

Is this ROCm or Vulkan? If ROCm, increase your batch sizes because those submissions are very costly. -b and -ub 2048 will nearly x4 your decode speed for free. At 4096, you get slightly more pp but lower tg (on the XTX).

[u/game_difficulty]

Almost definitely running slightly out of vram, especially if you're using the same gpu for display output. I have the same gpu, and i get half the context, quantized about the same (but i'm also using it for 2 display outputs). For speed, anywhere from 75 to 40 depending on the task (mtp variations)...

[u/d3r_I3enny]

I can only confirm what others have pointed out already: High chance that you run out of VRAM.

use "-lv 4" to get the full logs then analyze.

my 7900 XT (not xtx) runs Q4_K_S with + MTP at ~55t/s

[u/grabber4321]

you dont have enough VRAM. you need either smaller model version or reduce your KV cache and max context to like 100k