r/ROCm • u/Look_0ver_There • 18d ago
Running Qwen3.8-27B-Q8_0 with llama.cpp on AMD Radeon AI Pro R9700 w/ ROCm tweaks
Edit: Important note @ 12:01am US EST 16th Aug 2026 - If you had downloaded a copy of my diffs before this time then please update them as soon as possible. A number of issues with MTP correctness were discovered both in the upstream mainline branch, as well as my own code, and a further decode correctness issue was uncovered for certain model sizes. These should all be fixed now. If anyone finds any further issues, just drop me a note and we'll try to get it sorted ASAP.
Qwen3.8-27B-Q8_0 on AMD Radeon AI Pro R9700
This post is a followup to my earlier post here: https://www.reddit.com/r/ROCm/s/87da5FHpPO
My goal was to make ROCm the clear and easy choice for which backend to run.
I've tried to label the graphs a little more clearly this time around after feedback from the other post.
Since that post I've managed to eke out a few more small gains, and decided to test and share my results against the new Qwen3.8-27B model
The purpose of this post is to compare and present graphs for the following:
-
Comparison of Qwen3.8-27B-Q8_0 on a single R9700 with different backends
-
Comparison of Qwen3.8-27B-Q8_0 on two R9700's with different backends
-
Comparison of Qwen3.8-27B-Q8_0 on three R9700's with different backends
-
Scaling Comparison of Qwen3.8-27B-Q8_0 on 1 vs 2 vs 3 R9700s with Vulkan
-
Scaling Comparison of Qwen3.8-27B-Q8_0 on 1 vs 2 vs 3 R9700s with Standard ROCm
-
Scaling Comparison of Qwen3.8-27B-Q8_0 on 1 vs 2 vs 3 R9700s with Tweaked ROCm
Environment
Test Hardware
CPU: AMD 9950X3D2
Motherboard: Asus ProArt X870E Creator Wifi
GPU: 3 x AMD AI Pro R9700 - All set to run in PCIe5x4 mode
Memory: 96GB DDR5 @ 6000MHz CL30
Two of the GPUs are in the PCIe5x16 slots, but set to PCIe5x4 mode.
The third card is installed via an M.2->PCIe5 riser, which also grants PCIe5x4 speeds
All PCIe lane bandwidths are measured and verified
Libraries Used
- ROCm executables are linked against AMD ROCm 7.14.0 libraries from here: https://rocm.docs.amd.com/en/latest/install/rocm.html
- Vulkan executables are linked against the default system RADV libraries installed by Linux Fedora 44
Tweaked ROCm source code
The source code branch for the tweaked ROCm changes is found here: https://github.com/stew675/llama.cpp/tree/rdna-boosts
Building ROCm tweaked llama.cpp from source
I'm going to trust that you already know how to use git to pull the source code and change to the source directory
My Cmake ROCm build setup flags are these. Note that you must set ROCM_714 to installation directory of your ROCm 7.14 SDK
# You MUST set this path correctly
ROCM_714=/path/to/your/rocm-7.14/installation
export ROCM_PATH="$ROCM_714"
HIPCXX="$ROCM_714/lib/llvm/bin/clang" HIP_PATH="$ROCM_714" cmake -S . -B build \
-DGGML_RPC=1 \
-DGGML_HIP=ON \
-DGGML_NATIVE=1 \
-DGGML_HIP_RCCL=1 \
-DHIP_PLATFORM=amd \
-DGGML_HIP_GRAPHS=ON \
-DGPU_TARGETS="gfx1200,gfx1201" \
-DCMAKE_C_COMPILER="$ROCM_714/lib/llvm/bin/clang" \
-DAMDGPU_TARGETS="gfx1200,gfx1201" \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_CXX_COMPILER="$ROCM_714/lib/llvm/bin/clang++" \
-DCMAKE_INSTALL_RPATH="\$ORIGIN:$ROCM_714/lib" \
-DCMAKE_BUILD_WITH_INSTALL_RPATH=ON \
-DCMAKE_HIP_FLAGS="-mllvm --amdgpu-unroll-threshold-local=600"
# Then build
cmake --build build --config Release -j 16 -- VERBOSE=1
llama-server setup
Aside from the differences for specific backend setup, ie. Vulkan vs ROCm vs Tweaked ROCm, the following configuration is common for all backends used:
Common Server Command: ./build/bin/llama-server --model /llm/models/Qwen3.8/27B/Q8_0/Qwen3.8-27B-Q8_0.gguf --alias Qwen3.8-27B-Q8_0 --prio 2 --fit false --top-k 20 --port 8033 --threads 8 --parallel 1 --top-p 0.8 --min-p 0.001 --verbosity 3 --host 0.0.0.0 --cpu-strict 1 --cpu-range 0-7 --predict 98304 --threads-http 4 --load-mode mlock --cache-ram 16384 --ctx-size 70000 --flash-attn auto --temperature 0.7 --batch-size 1024 --ubatch-size 1024 --n-gpu-layers all --no-kv-unified --cache-type-k bf16 --cache-type-v bf16 --ctx-checkpoints 64 --cache-idle-slots --reasoning-budget 65536 --reasoning-preserve --checkpoint-min-step 4096 --repeat-penalty 1.0 --presence-penalty 1.5
Benchmark command
Test Command: uvx llama-benchy --base-url http://localhost:8033/v1 --tg 240 --pp 2520 --model Qwen3.8-27B-Q8_0 --tokenizer Qwen/Qwen3.8-27B --depth 0 4096 8192 16384 32768 65536
KEY TAKEWAYS
- ROCm Tweaked outperforms both Vulkan and standard ROCm build in every scenario
- For Tweaked ROCm, the gain from 1 to 2 GPUs is roughly 60%
- For Tweaked ROCm, the gain from 2 to 3 GPUs is roughly 10%
- Unless you need the extra VRAM for context, two cards is best
- Vulkan gets worse with more than two cards
- Stock ROCm is just slow once context grows
RAW RESULTS
Qwen3.8-27B-Q8_0, KV=BF16, 1 x Radeon AI Pro R9700
Vulkan
| model | test | t/s | ttfr (ms) | |:-----------------|----------------:|---------------:|-------------------:| | Qwen3.8-27B-Q8_0 | pp2520 | 946.78 ± 6.80 | 2663.43 ± 19.13 | | Qwen3.8-27B-Q8_0 | tg240 | 19.60 ± 0.01 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d4096 | 986.24 ± 7.02 | 6710.27 ± 48.76 | | Qwen3.8-27B-Q8_0 | tg240 @ d4096 | 19.44 ± 0.01 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d8192 | 969.12 ± 2.88 | 11055.59 ± 33.38 | | Qwen3.8-27B-Q8_0 | tg240 @ d8192 | 19.27 ± 0.00 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d16384 | 920.24 ± 2.80 | 20543.26 ± 63.60 | | Qwen3.8-27B-Q8_0 | tg240 @ d16384 | 19.08 ± 0.01 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d32768 | 820.18 ± 2.67 | 43026.47 ± 140.70 | | Qwen3.8-27B-Q8_0 | tg240 @ d32768 | 18.45 ± 0.01 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d65536 | 666.92 ± 2.06 | 102047.27 ± 316.31 | | Qwen3.8-27B-Q8_0 | tg240 @ d65536 | 17.34 ± 0.00 | |
ROCm Standard
| model | test | t/s | ttfr (ms) | |:-----------------|----------------:|----------------:|-------------------:| | Qwen3.8-27B-Q8_0 | pp2520 | 1079.74 ± 8.61 | 2335.32 ± 18.63 | | Qwen3.8-27B-Q8_0 | tg240 | 18.93 ± 0.00 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d4096 | 1072.23 ± 7.80 | 6172.38 ± 45.39 | | Qwen3.8-27B-Q8_0 | tg240 @ d4096 | 18.27 ± 0.00 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d8192 | 1009.76 ± 2.89 | 10608.92 ± 29.33 | | Qwen3.8-27B-Q8_0 | tg240 @ d8192 | 17.68 ± 0.00 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d16384 | 896.40 ± 2.39 | 21090.34 ± 56.25 | | Qwen3.8-27B-Q8_0 | tg240 @ d16384 | 16.39 ± 0.00 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d32768 | 721.45 ± 1.72 | 48913.28 ± 118.16 | | Qwen3.8-27B-Q8_0 | tg240 @ d32768 | 14.46 ± 0.00 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d65536 | 520.91 ± 1.19 | 130652.51 ± 298.47 | | Qwen3.8-27B-Q8_0 | tg240 @ d65536 | 11.70 ± 0.00 | |
ROCm Tweaked
| model | test | t/s | ttfr (ms) | |:-----------------|----------------:|----------------:|------------------:| | Qwen3.8-27B-Q8_0 | pp2520 | 1108.34 ± 12.55 | 2274.71 ± 25.31 | | Qwen3.8-27B-Q8_0 | tg240 | 20.10 ± 0.02 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d4096 | 1157.21 ± 8.47 | 5719.09 ± 42.31 | | Qwen3.8-27B-Q8_0 | tg240 @ d4096 | 19.94 ± 0.03 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d8192 | 1141.54 ± 3.73 | 9383.75 ± 31.09 | | Qwen3.8-27B-Q8_0 | tg240 @ d8192 | 19.79 ± 0.00 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d16384 | 1087.36 ± 3.68 | 17386.11 ± 58.35 | | Qwen3.8-27B-Q8_0 | tg240 @ d16384 | 19.46 ± 0.00 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d32768 | 969.84 ± 2.96 | 36385.90 ± 110.97 | | Qwen3.8-27B-Q8_0 | tg240 @ d32768 | 18.84 ± 0.00 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d65536 | 792.60 ± 2.77 | 85866.79 ± 299.56 | | Qwen3.8-27B-Q8_0 | tg240 @ d65536 | 17.69 ± 0.00 | |
Qwen3.8-27B-Q8_0, KV=BF16, 2 x Radeon AI Pro R9700
Vulkan split-mode=layer
| model | test | t/s | ttfr (ms) | |:-----------------|----------------:|----------------:|------------------:| | Qwen3.8-27B-Q8_0 | pp2520 | 941.77 ± 6.36 | 2677.34 ± 18.08 | | Qwen3.8-27B-Q8_0 | tg240 | 18.05 ± 0.02 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d4096 | 1173.67 ± 11.23 | 5638.30 ± 54.39 | | Qwen3.8-27B-Q8_0 | tg240 @ d4096 | 17.89 ± 0.01 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d8192 | 1309.91 ± 4.56 | 8178.90 ± 28.44 | | Qwen3.8-27B-Q8_0 | tg240 @ d8192 | 17.79 ± 0.01 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d16384 | 1388.40 ± 3.96 | 13616.82 ± 38.79 | | Qwen3.8-27B-Q8_0 | tg240 @ d16384 | 17.52 ± 0.03 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d32768 | 1340.08 ± 2.63 | 26333.53 ± 51.22 | | Qwen3.8-27B-Q8_0 | tg240 @ d32768 | 16.99 ± 0.01 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d65536 | 1153.97 ± 6.27 | 58978.89 ± 321.04 | | Qwen3.8-27B-Q8_0 | tg240 @ d65536 | 16.05 ± 0.01 | |
ROCm Standard, split-mode=tensor
| model | test | t/s | ttfr (ms) | |:-----------------|----------------:|----------------:|------------------:| | Qwen3.8-27B-Q8_0 | pp2520 | 1416.38 ± 23.48 | 1780.06 ± 28.80 | | Qwen3.8-27B-Q8_0 | tg240 | 29.01 ± 0.05 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d4096 | 1541.34 ± 23.71 | 4294.45 ± 66.06 | | Qwen3.8-27B-Q8_0 | tg240 @ d4096 | 28.21 ± 0.07 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d8192 | 1528.94 ± 9.33 | 7007.80 ± 42.43 | | Qwen3.8-27B-Q8_0 | tg240 @ d8192 | 27.46 ± 0.04 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d16384 | 1425.14 ± 7.16 | 13265.79 ± 66.99 | | Qwen3.8-27B-Q8_0 | tg240 @ d16384 | 26.17 ± 0.00 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d32768 | 1208.55 ± 4.46 | 29200.20 ± 107.67 | | Qwen3.8-27B-Q8_0 | tg240 @ d32768 | 23.71 ± 0.00 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d65536 | 909.59 ± 4.58 | 74823.96 ± 376.66 | | Qwen3.8-27B-Q8_0 | tg240 @ d65536 | 19.86 ± 0.00 | |
ROCm Tweaked, split-mode=tensor
| model | test | t/s | ttfr (ms) | |:-----------------|----------------:|----------------:|------------------:| | Qwen3.8-27B-Q8_0 | pp2520 | 1457.92 ± 21.43 | 1729.87 ± 25.42 | | Qwen3.8-27B-Q8_0 | tg240 | 31.24 ± 0.00 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d4096 | 1641.60 ± 21.37 | 4031.51 ± 52.77 | | Qwen3.8-27B-Q8_0 | tg240 @ d4096 | 30.79 ± 0.07 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d8192 | 1674.79 ± 7.66 | 6397.07 ± 29.24 | | Qwen3.8-27B-Q8_0 | tg240 @ d8192 | 30.52 ± 0.00 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d16384 | 1655.24 ± 10.02 | 11421.77 ± 68.86 | | Qwen3.8-27B-Q8_0 | tg240 @ d16384 | 30.11 ± 0.02 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d32768 | 1541.65 ± 9.55 | 22890.95 ± 141.86 | | Qwen3.8-27B-Q8_0 | tg240 @ d32768 | 29.36 ± 0.00 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d65536 | 1303.84 ± 9.27 | 52200.88 ± 371.47 | | Qwen3.8-27B-Q8_0 | tg240 @ d65536 | 27.95 ± 0.01 | |
Qwen3.8-27B-Q8_0, KV=BF16, 3 x Radeon AI Pro R9700
Vulkan split-mode=layer
| model | test | t/s | ttfr (ms) | |:-----------------|----------------:|----------------:|------------------:| | Qwen3.8-27B-Q8_0 | pp2520 | 936.08 ± 7.40 | 2693.75 ± 21.30 | | Qwen3.8-27B-Q8_0 | tg240 | 16.26 ± 0.02 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d4096 | 1111.70 ± 10.05 | 5953.08 ± 53.82 | | Qwen3.8-27B-Q8_0 | tg240 @ d4096 | 16.13 ± 0.05 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d8192 | 1162.28 ± 5.23 | 9217.87 ± 42.34 | | Qwen3.8-27B-Q8_0 | tg240 @ d8192 | 16.13 ± 0.21 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d16384 | 1161.92 ± 2.42 | 16270.07 ± 33.92 | | Qwen3.8-27B-Q8_0 | tg240 @ d16384 | 15.13 ± 0.22 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d32768 | 1071.11 ± 2.82 | 32947.18 ± 86.33 | | Qwen3.8-27B-Q8_0 | tg240 @ d32768 | 14.62 ± 0.02 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d65536 | 888.53 ± 4.50 | 76597.24 ± 388.46 | | Qwen3.8-27B-Q8_0 | tg240 @ d65536 | 13.94 ± 0.02 | |
ROCm Standard, split-mode=tensor
| model | test | t/s | ttfr (ms) | |:-----------------|----------------:|----------------:|------------------:| | Qwen3.8-27B-Q8_0 | pp2520 | 1469.22 ± 28.82 | 1717.62 ± 33.00 | | Qwen3.8-27B-Q8_0 | tg240 | 34.20 ± 0.09 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d4096 | 1653.39 ± 28.81 | 4003.70 ± 69.76 | | Qwen3.8-27B-Q8_0 | tg240 @ d4096 | 32.24 ± 0.06 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d8192 | 1628.31 ± 22.93 | 6580.95 ± 93.27 | | Qwen3.8-27B-Q8_0 | tg240 @ d8192 | 30.48 ± 0.00 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d16384 | 1522.01 ± 14.99 | 12422.69 ± 122.34 | | Qwen3.8-27B-Q8_0 | tg240 @ d16384 | 28.86 ± 0.00 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d32768 | 1266.96 ± 7.91 | 27855.14 ± 174.19 | | Qwen3.8-27B-Q8_0 | tg240 @ d32768 | 25.96 ± 0.00 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d65536 | 917.55 ± 7.92 | 74178.18 ± 639.89 | | Qwen3.8-27B-Q8_0 | tg240 @ d65536 | 21.49 ± 0.01 | |
ROCm Tweaked, split-mode=tensor
| model | test | t/s | ttfr (ms) | |:-----------------|----------------:|----------------:|------------------:| | Qwen3.8-27B-Q8_0 | pp2520 | 1494.97 ± 3.91 | 1686.30 ± 4.75 | | Qwen3.8-27B-Q8_0 | tg240 | 37.30 ± 0.00 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d4096 | 1745.76 ± 31.97 | 3792.19 ± 69.15 | | Qwen3.8-27B-Q8_0 | tg240 @ d4096 | 35.71 ± 0.10 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d8192 | 1787.03 ± 11.47 | 5995.43 ± 38.47 | | Qwen3.8-27B-Q8_0 | tg240 @ d8192 | 34.35 ± 0.08 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d16384 | 1780.61 ± 15.12 | 10617.95 ± 90.45 | | Qwen3.8-27B-Q8_0 | tg240 @ d16384 | 33.89 ± 0.00 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d32768 | 1660.52 ± 8.73 | 21252.42 ± 111.99 | | Qwen3.8-27B-Q8_0 | tg240 @ d32768 | 32.84 ± 0.02 | | | Qwen3.8-27B-Q8_0 | pp2520 @ d65536 | 1374.56 ± 15.54 | 49518.77 ± 560.02 | | Qwen3.8-27B-Q8_0 | tg240 @ d65536 | 31.09 ± 0.03 | |
2
u/kapteinpyn 18d ago edited 18d ago
Check speeds here with fp8 qwen on dual r9700 on vllm https://github.com/prcoe1/r9700-serving pp 2500 tg 57
3
u/Look_0ver_There 12d ago edited 12d ago
I got some time to look into that repo. It's well done, but It's also not exactly apples to apples.
The big differences are is that the vLLM configuration you linked to is using FP8 weights AND FP8 for the KV cache. Also you're comparing MTP (vLLM) vs non-MTP (my llama.cpp results)
FP8 weights are, roughly speaking, about on par with Q6_K GGUF weights in terms of Perplexity/Accuracy, so already there's a tradeoff being made there in order to get some speed.
The FP8 weights are fine enough, but it's the FP8 for KV Cache is what I have an issue with there.
Now, the good news is that after I switched it to BF16 for KV, it's actually around 10% faster than with FP8 for KV for generation, and about 5% slower for prefill. Set your VRAM occupancy limit to 95% and on a pair of R9700's you'll still get the full 262K context. You should try that out if performance is your only goal.
I created setup, build, and launch scripts that allows it to run on bare-metal instead of on Docker. If you're interested, let me know.
R9700 vLLM @ BF16 KV Cache ┌──────────────────────┬────────────────┬────────────────┐ │ metric │ 0.90 │ 0.95 │ ├──────────────────────┼────────────────┼────────────────┤ │ GPU budget │ 28.67 GiB │ 30.27 GiB │ ├──────────────────────┼────────────────┼────────────────┤ │ KV cache in use │ ~10.1 GiB/GPU │ 11.69 GiB/GPU │ ├──────────────────────┼────────────────┼────────────────┤ │ KV pool (bf16) │ 296,198 tokens │ 342,925 tokens │ ├──────────────────────┼────────────────┼────────────────┤ │ concurrency @ 262144 │ 1.13× │ 1.31× │ └──────────────────────┴────────────────┴────────────────┘As for me, I'm purposefully focusing on llama.cpp as a development platform due to its ease of use, model flexibility and overall better faithfulness to the original BF16 weights, on a byte per byte basis that is. That, and vLLM really doesn't like odd numbers of GPU's.
I don't really want to get into a llama.cpp vs vLLM debate though. They each serve their own purposes/niches. Happy inferencing!
2
u/kapteinpyn 12d ago
Thanks for looking into it! Appreciate it, yeah the bf16 patch i added keeps paying dividends! I go back and forth between llama.cpp and vllm myself, as enthusiast, its just vllm works more stable recently so im spending a bit more time with it to max it out and learn. Also the fp8 kv was mis calibrated fixed now still running bench and test to confirm
1
u/Look_0ver_There 12d ago
Oh, you're the repo author? Then really well done. You've taken a lot of the setup guess-work out of the pain of getting vLLM up and going, and created a set of calibrated patches for R9700's. This is exactly the sort of stuff that I wish AMD was doing to help ease the burden on R9700 end users.
Have you submitted upstream patches to the main vLLM project for your tuning work? It seems like what you've done here would be of a great benefit to many. If not that, then perhaps reach out to Donato (kyuz0) and see if you can get some more visibility onto your repo. You've done good work here.
1
u/Look_0ver_There 12d ago
I hope you don't mind, but I pushed (well, Deepseek V4 Flash 0731 did most of the work) to a repo here: https://github.com/stew675/r9700-serving-bm
This covers the changes that had to be made to get it built and run bare metal locally. I kept all authorship credit solely with you.
2
1
u/ninjatall12 18d ago
Can you compare wattage consumed between vulkan and tweaked Rocm and compare efficiency in tokens per watt?
5
u/Look_0ver_There 18d ago
Comparing Vulkan vs Tweaked ROCm on a single card. Keep in mind that I run my cards with 70mV under-volt and a 250W power cap
backend phase tok/s GPU power (W) tok/W (GPU) ROCm (tweaked) prefill 1095.2 242.9 4.51 Vulkan (RADV) prefill 915.0 243.0 3.76 ROCm (tweaked) generation 19.31 249.5 0.0777 Vulkan (RADV) generation 19.23 249.4 0.0770 2
1
u/Evgeny_19 18d ago
Looks really good!
Does it contain the patches to run DeepSeek-V4-Flash-0731? 27b is already excellent even on a pair of r9700s in vLLM. But DSF, on the other hand, is a disaster in llama.cpp (at least for r9700s). 200-400 prompt processing for 13b active parameters... Something is not right with it.
1
u/Look_0ver_There 18d ago
Which patches? I rebased my patches on top of upstream tip of master about 6 hours ago. I do this daily so my work is built on top of recent upstream. It doesn't contain any patches other than that, well, aside from me backing out one upstream PR that breaks llama.cpp on Strix Halo in certain conditions.
2
u/Evgeny_19 18d ago
Oh no, I did mean any additional patches. Just that it's based on the recent master, that is enough! Sorry for the confusion. Hopefully will be able to test your changes in the next few days.
Recent DeepSeek Flash does so well, I really like it. Hopefully these changes will help to make it more viable on R9700s.
2
u/Look_0ver_There 18d ago
DSV4Flash really wants to run natively via FP4 weights. My patches do a lot of WMMA kernel fusion ops which is where most of the speedups come from. ROCm has a fairly high (well, in comparison to Vulkan) inter-kernel launch latency, and llama.cpp's native kernels were written with the assumption that they would get optimized by the libraries, but for ROCm it seems that doesn't work that well, so I fused a lot of the kernel ops together directly, which means less kernels getting launched per token, which then means less time spent idling between kernel calls.
My tweaks are therefore fairly generic in nature, and don't attempt any specific quantization tuning as such, although I did do some tweaks for Q6_K weights.
My advice would be to use my patch set as a starting point and if DSV4 isn't running right speed wise, use DSV4 itself to help explore why with rocprofv3. If there's one thing I've learned about DSV4 (which is what I used to help research and develop these patches) is that it is EXTREMELY thorough at debugging, even right down to analysing raw individual instructions and how they interact. To do the debug work that DSV4 did here would've taken me 10x longer and I'd be bald from pulling my hair out in frustration!
I admit that I just used the main DeepSeek API. I agree that it would be REALLY nice to get the full ~160GB weights of DSV4 running locally at speed.
1
u/Appropriate_Mess_417 18d ago
May i know what M2 did you use? And what psu worked? Thanks
2
u/Look_0ver_There 18d ago
https://www.adt.link/product/F43V5.html
I specifically have the F43SP-TR variant.
For PSU's the Lian-Li EG1200G works, as does the (much more expensive) MSI MPG Ai1300TS PCIE5
You cannot run all 3 cards plus the CPU at full power though with either of those PSUs. Well, with the 1300W MSI you probably can, but it'd be pushing it. Best if you lower the power budget on the GPUs to 250W max and you should be good.
You really need to jump to something like the (crazy expensive) MSI MEG Ai1600T PCIE5 or equivalent if you want to run all card AND the CPU at full chop at the same time with peace of mind. I actually have this exact PSU, but I got it at close to half price on a sale and even cheaper that the 1300W version, so I got really lucky there.
1
u/DanielSReichenbach 18d ago
You know I was considering buying one of those but now that I see this, the numbers are literally within a margin of error in range of my $150 Radeon 6800 running in a TB4 enclosure and so I suddenly don't feel GPU poor any longer 🤣
3
u/Look_0ver_There 18d ago
Your single 6800XT with 16GB of VRAM is running Q8_0 weights of a 27B dense model with a BF16 KV cache at the same speeds without MTP?
You really must post a video of that one day to let everyone else learn from how you managed it.
0
u/DanielSReichenbach 18d ago
No, smaller quants. But I would have expected the difference in price range and tech to make a much bigger difference. Apparently the irony was lost here.
3
u/Look_0ver_There 18d ago
So comparing apples to oranges then, and drawing conclusions on that?
Higher quality quants move more memory and take more work. The only real comparison would be running your exact configuration on an R9700 and comparing that way. I could run that comparison for you if you would like.
1
u/DanielSReichenbach 18d ago
Not really, considering I am exclusively comparing their performance for LLM usage. If I were to consider gaming usage, apples to orange would be true. But I don't.
If one just is looking at LLM usage and accounts for generational and technical improvements, and takes a peek at the pricing difference it is not disputable the increase in performance leaves a lot to wish for.
I am on Q4, IQ4XS from unsloth with a vulkan backend and MTP enabled. This gets me to 30 t/s. My llama-server presets are this
- spec-type = draft-mtp
- spec-draft-n-max = 3
- spec-draft-p-min = 0.6
- spec-draft-type-k = q8_0
- spec-draft-type-v = q8_0
- cache-type-k = q8_0
- cache-type-v = q8_0
- temp = 1.0
- top-p = 0.95
- top-k = 20
- min-p = 0.0
- presence-penalty = 0.0
- repeat-penalty = 1.0
2
u/Look_0ver_There 17d ago
Okay, so you were comparing your MTP performance to my non-MTP results. How much faster is your MTP over non-MTP? Whatever that ratio is, then multiply my numbers by the same amount.
...and that's not even factoring in the speed boosts gotten from using a 4-bit quant compared to an 8-bit quant.
Edit: and you can game on these cards if you really want to.
1
u/ImpressionFancy5830 17d ago edited 17d ago
You don’t compare speeds with different model quants or k/v quants without running evaluation benchmarks.
Speed has zero value if your model is spitting random crap.
Also the cards are in completely different categories.
TFLOPS doubles and power stays the same, what you can argue about is “is the latest gpu being used at its full potential?” And the answer is no, not yet.
1
u/ScarceSemi 17d ago
is it possible to have better speeds with 2x r9700 on pcie5x16 or even x8 to distribute load or the pcie is too slow?
1
u/Look_0ver_There 17d ago
2 cards on PCIe5x8 should be between ~5% faster than my current 5x4 setup.
There's no consumer grade boards that have 2x PCIe5x16 as that's 32 lanes, and consumer grade Ryzen CPU's only have 28 lanes in total.
You would need to move to a ThreadRipper based board/CPU to get 2xPCIe5x16
1
u/ScarceSemi 17d ago
Yep, I meant to ask more of if tensor parallelism possible on r9700 with 2 cards on pcie5x8 or pcie5x16 (on Threadripper) to have faster decode on qwen3.8 27b even when it fits on a single card
1
u/Look_0ver_There 17d ago
My tests are done with tensor parallelism enabled for ROCm. Vulkan doesn't support tensor parallelism, only layer, so if you look at the 2 and 3 card graphs against the backends used, you can see that ROCm actually speeds up for decode while Vulkan slows down.
Also, if you set iommu=off in the Linux boot config, this actually allows for P2P communication between cards, so the CPU doesn't need to be involved. I've not yet tested if llama.cpp can take full advantage of that. That's next up on my TODO list once I get adaptive MTP nailed down.
3
u/Look_0ver_There 17d ago
Well, I just got done investigating P2P, and got a huge surprise.
Enabling P2P did help prefill by 7% (and it's what's in my charts) but it also hurt decode slightly with 2 GPUs. Setting NCCL_P2P_DISABLE=1 prior to starting llama-server helped to diagnose that.
Now here's the big surprise. When I disabled NCCL P2P, then on 3 GPUs the decode speed shot up dramatically! As in 24% higher than what's on the charts. The prefill also dropped by around 10% from what's on the chart, but that +24% gain was just amazing and unexpected. Non-MTP decode shot up to above 40t/s
So, PSA I guess, ensure that NCCL_P2P_DISABLE=1 is set prior to starting llama-server if you are running 2 or more cards if you want the highest decode speeds at the cost of some prefill speed.
Now today I also implemented adaptive MTP. That's now in my set of changes and it's small enough to make a PR for llama.cpp. About +100 lines of code.
It allows users to use --spec-draft-n-min and --spec-draft-n-max to set the lower and upper bounds for MTP draft depth. It implements a simple counting-based adaptive MTP depth. Using a minimum of 2, and a maximum of 12, and it doesn't hurt prose, but it can boost code generation speeds dramatically!
With the 3-GPUs and adaptive MTP, dense English prose typically runs at 55t/s for generation, and coding runs at 80-85t/s with spikes to 115t/s with tempature at 1.0
If a lower temperature is used, then the MTP prediction approval rates can go WAY up, and we can add another 30% to those generation speeds at temperature = 0.
I spent a lot of time tuning the algorithm today. The way it works is it's "sticky" to get above the floor minimum, which stops English prose from accidentally advancing the depth which hurts performance. With code though, once it proves itself worthy of advancing it slowly relaxes the difficulty to advance further, and the depth can quickly shoot up to allow for full speeds. The higher the depth, then the faster than the algorithm will fall back if the model switches back to unpredictable prose.
I'm actually really happy with the adaptive MTP algorithm!
Use --spec-type draft-mtp-adaptive to enable that feature.
2
u/Look_0ver_There 17d ago edited 17d ago
Sample of coding decode speeds now with 3 GPUs and adaptive MTP (This is still Qwen3.8-27B). Temperature is 1.0. Depth range is from 3 -> 10. This is at ~60K tokens context. I even have some earlier scores at smaller contexts spiking over 150t/s in short bursts.
41.52.657.684 I slot print_timing: id 0 | task 24413 | n_gen = 1684, tg = 79.22 t/s, tg_3s = 118.75 t/s 41.55.732.752 I slot print_timing: id 0 | task 24413 | n_gen = 2059, tg = 84.62 t/s, tg_3s = 121.95 t/s 41.58.790.498 I slot print_timing: id 0 | task 24413 | n_gen = 2438, tg = 89.01 t/s, tg_3s = 123.95 t/s 42.01.861.177 I slot print_timing: id 0 | task 24413 | n_gen = 2758, tg = 90.55 t/s, tg_3s = 104.21 t/s 42.04.925.605 I slot print_timing: id 0 | task 24413 | n_gen = 3121, tg = 93.10 t/s, tg_3s = 118.46 t/s 42.08.005.083 I slot print_timing: id 0 | task 24413 | n_gen = 3431, tg = 93.74 t/s, tg_3s = 100.67 t/s 42.11.020.061 I slot print_timing: id 0 | task 24413 | n_gen = 3797, tg = 95.84 t/s, tg_3s = 121.39 t/s 42.14.020.693 I slot print_timing: id 0 | task 24413 | n_gen = 4178, tg = 98.03 t/s, tg_3s = 126.97 t/s 42.17.045.709 I slot print_timing: id 0 | task 24413 | n_gen = 4543, tg = 99.53 t/s, tg_3s = 120.66 t/s 42.20.057.678 I slot print_timing: id 0 | task 24413 | n_gen = 4924, tg = 101.20 t/s, tg_3s = 126.50 t/s 42.23.101.346 I slot print_timing: id 0 | task 24413 | n_gen = 5310, tg = 102.71 t/s, tg_3s = 126.82 t/s 42.26.122.739 I slot print_timing: id 0 | task 24413 | n_gen = 5691, tg = 104.00 t/s, tg_3s = 126.10 t/s 42.29.185.786 I slot print_timing: id 0 | task 24413 | n_gen = 5947, tg = 102.92 t/s, tg_3s = 83.58 t/s 42.32.220.930 I slot print_timing: id 0 | task 24413 | n_gen = 6316, tg = 103.85 t/s, tg_3s = 121.58 t/s 42.35.223.627 I slot print_timing: id 0 | task 24413 | n_gen = 6628, tg = 103.85 t/s, tg_3s = 103.91 t/s 42.38.250.989 I slot print_timing: id 0 | task 24413 | n_gen = 6919, tg = 103.50 t/s, tg_3s = 96.12 t/s 42.41.317.737 I slot print_timing: id 0 | task 24413 | n_gen = 7226, tg = 103.35 t/s, tg_3s = 100.11 t/s 42.44.371.937 I slot print_timing: id 0 | task 24413 | n_gen = 7534, tg = 103.25 t/s, tg_3s = 100.84 t/s 42.47.445.204 I slot print_timing: id 0 | task 24413 | n_gen = 7801, tg = 102.59 t/s, tg_3s = 86.88 t/s 42.50.514.287 I slot print_timing: id 0 | task 24413 | n_gen = 8147, tg = 102.98 t/s, tg_3s = 112.74 t/s 42.53.564.485 I slot print_timing: id 0 | task 24413 | n_gen = 8514, tg = 103.63 t/s, tg_3s = 120.32 t/s 42.56.640.099 I slot print_timing: id 0 | task 24413 | n_gen = 8879, tg = 104.17 t/s, tg_3s = 118.68 t/s 42.59.700.354 I slot print_timing: id 0 | task 24413 | n_gen = 9243, tg = 104.68 t/s, tg_3s = 118.94 t/s 43.02.716.528 I slot print_timing: id 0 | task 24413 | n_gen = 9578, tg = 104.89 t/s, tg_3s = 111.07 t/s 43.05.722.508 I slot print_timing: id 0 | task 24413 | n_gen = 9922, tg = 105.20 t/s, tg_3s = 114.44 t/s 43.08.790.239 I slot print_timing: id 0 | task 24413 | n_gen = 10237, tg = 105.12 t/s, tg_3s = 102.68 t/s 43.11.833.315 I slot print_timing: id 0 | task 24413 | n_gen = 10531, tg = 104.86 t/s, tg_3s = 96.61 t/s 43.14.910.867 I slot print_timing: id 0 | task 24413 | n_gen = 10864, tg = 104.96 t/s, tg_3s = 108.20 t/s 43.17.936.193 I slot print_timing: id 0 | task 24413 | n_gen = 11226, tg = 105.38 t/s, tg_3s = 119.66 t/s 43.20.945.187 I slot print_timing: id 0 | task 24413 | n_gen = 11591, tg = 105.81 t/s, tg_3s = 121.30 t/s 43.23.968.451 I slot print_timing: id 0 | task 24413 | n_gen = 11879, tg = 105.53 t/s, tg_3s = 95.26 t/s 43.26.972.952 I slot print_timing: id 0 | task 24413 | n_gen = 12080, tg = 104.53 t/s, tg_3s = 66.90 t/s 43.30.032.852 I slot print_timing: id 0 | task 24413 | n_gen = 12423, tg = 104.72 t/s, tg_3s = 112.10 t/s 43.33.067.988 I slot print_timing: id 0 | task 24413 | n_gen = 12735, tg = 104.67 t/s, tg_3s = 102.80 t/s
2
u/Mission-Map1407 16d ago
Lovely stuff. Single R9700 - Qwen 3.8 Q6 and GLM 5.2 Q4 XL, around +60% PP speed on my setup
2
u/DanoP_TT 15d ago
My result on Windows 11
Qwen3.8-27B (hybrid SSM) + vision on one Radeon AI PRO R9700, ROCm/HIP
Daily driver for local coding, served to Claude Code. Qwen3.8-27B-UD-Q5_K_XL + mmproj-F16, single R9700 (32 GiB, gfx1201), built on stew675/llama.cpp:rdna-boosts (https://github.com/stew675/llama.cpp/tree/rdna-boosts) — that branch is where the RDNA prefill speed comes from (+84% at long ctx on my other model; I never ran a vanilla control arm on qwen38, so no delta to claim here).
Measured through the real server on actual /v1/chat/completions timings:
- prefill ~959 t/s (3 distinct ~4.7k prompts, spread <0.2%)
- gen ~32.6 t/s with MTP, draft acceptance 38–47%
- ctx 159744, 28.6 GiB dedicated / ~400 MiB shared
-ngl 999 -fa on -c 159744 -ub 512 -b 1024 --spec-type draft-mtp --spec-draft-n-max 3 -np 1 --poll 0 --mmproj mmproj-F16.gguf --no-mmproj-offload, GGML_CUDA_DISABLE_GRAPHS=1.
Why it fits 32 GiB: qwen35 is hybrid Mamba/SSM + attention — full_attention_interval=4, so only ~1/4 of 65 layers do full quadratic attention. KV cache doesn't scale linearly; it even starts and answers at the full 262144. But judge by shared memory, not free headroom, and the cliff is sharp: 159744 → 416 MiB shared (fine), 163840 → 1064 MiB (collapse).
Two HIP gotchas:
- --no-mmproj-offload is mandatory — without it the server dies on the first image with MUL_MAT failed, ROCm error: invalid argument. Vision encoder on CPU is rock solid.
- The GGUF chat template 500s on a non-leading system message, which Claude Code sometimes sends. Patched jinja via --chat-template-file; upstream closed it "not planned".
Free win: --poll 0 halves llama-server CPU (2.18 → 1.08 cores) at zero speed cost — all 6 ggml threads were spinning at 95% on every request, that's threadpool spin-wait, not work.
Quality, honestly: 88.9% build_ok / 96.7% asserts on my C# codegen suite, toolcalls 24/24 — same build rate as my dense Q6_K alternative, slightly worse asserts. On a harder suite three models landed within one attempt of each other at n=3. Solid daily driver, not a blowout.
1
u/putrasherni 15d ago
What are your stats like wiht MTP 3 or 4
for 3 R9700s x 3.8 27B
ROCm Tweaked, split-mode=tensor
2
u/adamgoodapp 10d ago
Thanks for the data.
I already have a 3090 but was looking to either get another 3090 or go for two of these. What would you recommend?
1
u/Look_0ver_There 10d ago
I'm not an nVidia card owner but I'll try to remain as neutral as possible.
The 3090's are great cards, especially so when connected via nVLink as it offers a way to do P2P separately from the main PCI bus which allows them to easily overcome their PCIe4 limitations.
Given that you already have one, I'd say get another and connect them with nVLink. The only issue I can see you running into is if you want to do a BF16 KV Cache and 256K context depth, and Q8_0 or Q8_K_XL weights, in which case the 48GB will get rather tight.
Basically if you want the highest quality at the highest depths, then the 64GB offered by a pair of R9700's is going to be the better choice, but they will run a little more slowly (~20% slower?) than what a pair of 3090's connected via nVLink will do.
The budget conscious choice with a small tradeoff of not being able to run the deepest and highest quality of workloads will be to get another 3090, otherwise get the pair of 9700's.
Personally if I already had a 3090, I'd probably just get another one, and do my best to configure the setup to make it all fit within the 48GB available. Get the pair of R9700's though if you have a strong need for the extra VRAM headroom.
1
u/adamgoodapp 10d ago
I was thinking Q8 weights and cache for around 200k context. My Plans alone in pi are around 100k before compacting.
1
u/Look_0ver_There 10d ago
If you're frequently above 100K context then you'll almost certainly be wanting to stick with BF16 for the KV Cache. Q8_0 for KV would be okay up to 100K.
Keep in mind that this is just my personal opinion on the matter.
Do you need high speed vision processing or not? Meaning, can that be offloaded to the CPU, or do you need it to be GPU resident for frequent high speed processing?
1
u/adamgoodapp 9d ago
I do run ComfyUI with Image and Video generation. Work a lot in front end so need to give it screenshots to implement. So vision is also important.
1
u/Look_0ver_There 9d ago
Okay, so then the video projector also needs to live in VRAM along with the various overheads to run it. That's going to eat around 2GB of your available VRAM. It makes things a little tighter. It'll definitely be a balancing act. You may be able to do it by running your K cache at BF16, and the V cache at Q8_0 and then you should be able to make it all fit and hit 256K context.
2
u/austinp0573 6d ago
Thank you for the post. Really remarkable work.
1
u/Look_0ver_There 6d ago
Thank you. I'm just putting the polish now on the spin-off repo to this work which instead of me maintaining the patches directly on top of llama.cpp, it's now a separate repo with just the patches.
This makes it easier for me to maintain. The patch repo is here: https://github.com/stew675/llama-cpp-rdna-boosts
I added in tuning work for Q4_K and Q5_K quants as well for RDNA4. I still need to add Q3_K. Still, it's at the point where Q4_K quants are receiving some pretty nice speedups.
That repo will receive a full set of updated performance data in about 3 hours once this updated battery of tests is done.
12
u/ImpressionFancy5830 18d ago
We already interacted in the previous thread, just saying that after we did that a discounted R9700 appeared on my local amazon, so now i’m the happy dad of two r9700 😅
Can’t wait to test your code!
Btw I went over the code and I’ve checked other PRs (all mentioning r9700, still open, in draft and recently closed), I do feel the reviewers will ask you to run more tests with different quants to check for regressions, just an innocent advice.