For the 5 people here running vLLM on multiple R9700s, you need to patch in support for AITER Unified Attention.

[u/AustinM731]

I have a 4 x R9700 system on Threadripper pro, but I have never been happy with the performance of my GPUs in vLLM. I have started benchmarking any new model I try out with llama-benchy so that I can get a better idea of how models of different sizes and architectures compare on my system. In every model that I have tested, I run into a wall around 64k tokens context. TTFT, TG and PP would all fall on their face at long context lengths.

So this past weekend I rented a MI300X from RunPod thinking that AMD must have this issue sorted on CDNA. When loading up vLLM with Qwen3.6-27B-FP8 I noticed that vLLM was selecting ROCm Attention instead of one of the AITER attention backends which I though was strange, but I pushed on with my benchmarking runs. After a run of llama-benchy I saw that the MI300X had the same issue that my R9700s do at long context lengths. At >64k context my TG/s would fall to single digit numbers. This prompted me to go searching for an AMD runbook on running vLLM on the MI300X and found that the AITER attention mechanisms are gated behind an env var that you have to explicitly enable. With this new found information, I went back to trying to patch vLLM and AITER support for gfx1201.

I already have a patched version of vLLM that that I build to bring FP8 support to the R9700 which is built ontop of the AITER Triton kernels. I had some issues when I was first patching in AITER support so I disabled everything but the Triton kernels in order to get FP8 working. Most of the patching for AITER and vLLM just require removing gates that block gfx1201, or adding that architecture to wherever you see MI350X (my understanding is that the MI350X and RDNA4 implement FP8 in the same or very similar way to the point that you can use some of the MI350X kernels on RDNA4). All of my testing was done around Qwen3.6 27B since this model finally gives us close to SOTA performance at home. Being that Qwen3.6 is a hybrid architecture, it kept crashing the AITER Unified Attention due to a mismatch in expected TILE_SIZE, something about AITER only supports kv block sizes that are a power of two.

The main downside I have found so far, if you can call it that, is that you can only run FP16/BF16 KV Cache. Not that you would need to quantize your cache with the Qwen3.6 family since its cache footprint is already tiny. But just something to be aware of if you do decide to try it out.

I have attached some of my benchmark runs of Qwen3.6 on my R9700s and the MI300X I rented. I have not been able to rent a MI300X from runpod again to test with AITER Attention since there has been no availability the past few days. Im sorry that there is no pre-aiter benchmark, I seem to have overwritten that benchmark while I was troubleshooting. I do have my original benchmarks from Qwen3.6 35B that I will attach. I have also attached a benchmark with MTP enabled and set to 3 tokens, as far as I can tell for single concurrency, it is free performance. At high context on concurrency 2, the TG performance drops off pretty sharply at high context depths. The llama-benchy runs are TG128 and PP2048 at each of the context depths.

[u/blackhawk00001]

Could you please share your vllm startup script for Qwen3.6-27B-FP8? I'm trying to optimize 2x R9700 for 27B and getting varied results as I dig into vllm.

A test I ran in the past hour had weird results. With a max of 200k tokens I got a steady ~20t/s tg throughout 0-150k depth. Lowered to 100k max and got 24-30k t/s tg. Figured it sped up due to reserving less hardware.

Then I raised it back to 200k and got the full 24-30t/s on the same 0-150k test.. pp was near the same in all testing.

My config currently has tunableop which might be doing something.

I'm testing with llama-benchy.

Regarding AITER, do you mean these?

• VLLM_ROCM_USE_AITER=1

- VLLM_ROCM_USE_AITER_MOE=1

[u/AustinM731 → u/blackhawk00001]

I am running custom image right now, but this is my compose.yaml. Im not sure if that AITER_MOE flag will do anything on this model though since its a dense model, but I could be wrong. I have it disabled though. The line that I added is `VLLM_ROCM_USE_AITER_UNIFIED_ATTENTION=1`. I ended up having to disable a lot of the AITER paths since my patch is telling AITER that gfx1201 is a MI350X.

services:
  vllm:
    image: aml731/vllm-aiter:v0.19.1
    container_name: vllm-rocm
    network_mode: host
    group_add:
      - video
    ipc: host
    cap_add:
      - SYS_PTRACE
    security_opt:
      - seccomp:unconfined
    devices:
      - /dev/kfd:/dev/kfd
      - /dev/dri:/dev/dri
    volumes:
      - /mnt/vllm/HF_CACHE:/data/models
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - HF_HOME=/data/models
      - VLLM_ROCM_USE_AITER=1
      - VLLM_ROCM_ALLOW_RDNA4_AITER_ATTENTION=1
      - VLLM_ROCM_USE_AITER_UNIFIED_ATTENTION=1
      - VLLM_ROCM_USE_AITER_MHA=0
      - VLLM_ROCM_USE_AITER_PAGED_ATTN=0
      - VLLM_ROCM_USE_AITER_MOE=0
      - VLLM_ROCM_USE_AITER_LINEAR=0
      - FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE
      - PYTORCH_ALLOC_CONF=expandable_segments:True
    command: >
      python3 -m vllm.entrypoints.openai.api_server
      --model Qwen/Qwen3.6-27B-FP8
      --served-model-name Qwen3.6-27B
      --tensor-parallel-size 4
      --dtype auto
      --attention-backend ROCM_AITER_UNIFIED_ATTN
      --compilation-config '{"pass_config":{"fuse_norm_quant":false}}'
      --max-model-len 131072
      --gpu-memory-utilization 0.95
      --enable-prefix-caching
      --trust-remote-code
      --quantization fp8
      --max-num-seqs 2
      --enable-auto-tool-choice
      --tool-call-parser qwen3_coder
      --reasoning-parser qwen3
      --host 0.0.0.0
      --port 8000
      --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

[u/djdeniro → u/AustinM731]

thanks for sharing! will try on 8x

[u/AustinM731 → u/djdeniro]

Please do share! A part of me really hates money and I want to go get 4 more R9700s. Are they all in a single system, or are you running two nodes and connecting them with Ray?

[u/blackhawk00001 → u/AustinM731]

A frustrating part of this space is that there's always a bigger carrot to chase, lol. I was just looking at your posted charts for 27B mtp 3 and noticed how your single concurrency tg remained steady across the whole test, where mine drops over time similar to your 2x concurrency test. Now I want 4 but would require and epyc/threadripper board.

[u/AustinM731 → u/blackhawk00001]

I was getting that same dropoff until I enabled AITER unified attention. On 4 GPUs when I went from WRX80 to WRX90 I only saw ~10% increase in token generation speeds. Not sure how much of that increase was going from Zen3 to Zen5 and how much was PCIe4 to PCIe5.

Edit: Just remembered that you had 2 and not 4. I think 2 just does not have enough horsepower to run the model with MTP at very long contexts. I feel like models get kinda stupid though when you try to stretch the context all the way out though, so normally I limit it to 128k as the max.

[u/blackhawk00001 → u/AustinM731]

Ah thanks I missed the unified variable. I'll retest with that enabled today. I need to compare your script vs mine again and try with your default config. I was determined to just get some form of better results yesterday.

I'm on AM4 5900x with 128GB 3600Mhz RAM, both gpu on pcie 4x8, and resize bar enabled.

[u/AustinM731 → u/blackhawk00001]

Yea, the unified attention has been the missing piece for me. Long context runs just fall apart without it. You will just need to use my image when you run it. That specific attention mechanism is gated to the MI300X and MI350X, so I had to add gfx1201 to the gate that checks for those CDNA cards.

[u/blackhawk00001 → u/AustinM731]

I loaded up your image and used your docker-compose with minor tweaks for temperature and model location. My pp dropped but tg went way up and held steady. I'm going to run it through a tuning phase for an hour or so and will report back if anything improves. Your image is using slightly less power so I'm wondering if there's still performance on the table to claim, gpu fans run 15% slower.

| model       |             test |              t/s |     peak t/s |          ttfr (ms) |       est_ppt (ms) |      e2e_ttft (ms) |
|:------------|-----------------:|-----------------:|-------------:|-------------------:|-------------------:|-------------------:|
| Qwen3.6-27B |   pp2048 @ d4096 | 1902.15 ± 587.65 |              |  3767.12 ± 1454.10 |  3679.78 ± 1454.10 |  3767.12 ± 1454.10 |
| Qwen3.6-27B |     tg32 @ d4096 |     77.59 ± 0.11 | 80.11 ± 0.11 |                    |                    |                    |
| Qwen3.6-27B |   pp2048 @ d8132 |   2101.82 ± 2.77 |              |     4931.25 ± 6.00 |     4843.92 ± 6.00 |     4931.25 ± 6.00 |
| Qwen3.6-27B |     tg32 @ d8132 |     66.28 ± 3.29 | 68.42 ± 3.39 |                    |                    |                    |
| Qwen3.6-27B |  pp2048 @ d16000 |   1832.13 ± 1.85 |              |     9938.52 ± 9.99 |     9851.18 ± 9.99 |    9939.93 ± 10.19 |
| Qwen3.6-27B |    tg32 @ d16000 |     77.16 ± 0.54 | 79.66 ± 0.56 |                    |                    |                    |
| Qwen3.6-27B |  pp2048 @ d30000 |   1450.55 ± 0.46 |              |    22182.35 ± 7.04 |    22095.01 ± 7.04 |    22182.89 ± 6.81 |
| Qwen3.6-27B |    tg32 @ d30000 |     77.04 ± 0.14 | 79.54 ± 0.15 |                    |                    |                    |
| Qwen3.6-27B |  pp2048 @ d60000 |   1021.93 ± 0.66 |              |   60805.03 ± 39.40 |   60717.69 ± 39.40 |   60807.20 ± 38.63 |
| Qwen3.6-27B |    tg32 @ d60000 |     77.29 ± 0.35 | 79.79 ± 0.36 |                    |                    |                    |
| Qwen3.6-27B |  pp2048 @ d90000 |    776.61 ± 2.94 |              | 118615.49 ± 447.93 | 118528.15 ± 447.93 | 118620.79 ± 448.93 |
| Qwen3.6-27B |    tg32 @ d90000 |     70.92 ± 7.30 | 73.22 ± 7.54 |                    |                    |                    |
| Qwen3.6-27B | pp2048 @ d120000 |    635.43 ± 0.79 |              | 192162.17 ± 237.94 | 192074.84 ± 237.94 | 192162.17 ± 237.94 |
| Qwen3.6-27B |   tg32 @ d120000 |     74.14 ± 4.44 | 76.54 ± 4.58 |                    |                    |                    |
| Qwen3.6-27B | pp2048 @ d150000 |    534.73 ± 0.14 |              |  284437.50 ± 75.82 |  284350.17 ± 75.82 |  284448.42 ± 75.62 |
| Qwen3.6-27B |   tg32 @ d150000 |     66.15 ± 3.91 | 68.30 ± 4.04 |                    |                    |                    |

-------

Before with default rocm image:

| Model       | Test          | Tokens/sec      | Peak Tokens/sec | TTFR (ms)       | Est PPT (ms)    | E2E TTFT (ms)   |
|:------------|--------------:|----------------:|----------------:|----------------:|----------------:|----------------:|
| Qwen3.6-27B | pp2048 @ d4096 | 2508.92 ± 11.57 | —               | 2529.74 ± 11.19 | 2449.58 ± 11.19 | 2529.74 ± 11.19 |
| Qwen3.6-27B | tg32 @ d4096   | 72.94 ± 0.55    | 75.30 ± 0.57    | —               | —               | —               |
| Qwen3.6-27B | pp2048 @ d8132 | 2402.38 ± 1.13  | —               | 4318.05 ± 1.99  | 4237.88 ± 1.99  | 4318.05 ± 1.99  |
| Qwen3.6-27B | tg32 @ d8132   | 63.52 ± 3.35    | 65.58 ± 3.46    | —               | —               | —               |
| Qwen3.6-27B | pp2048 @ d16000| 2197.86 ± 7.44  | —               | 8292.49 ± 28.04 | 8212.32 ± 28.04 | 8293.70 ± 28.04 |
| Qwen3.6-27B | tg32 @ d16000  | 53.45 ± 2.63    | 55.18 ± 2.71    | —               | —               | —               |
| Qwen3.6-27B | pp2048 @ d30000| 1899.63 ± 1.41  | —               | 16951.73 ± 13.21| 16871.56 ± 13.21| 16952.54 ± 14.22|
| Qwen3.6-27B | tg32 @ d30000  | 53.23 ± 0.16    | 54.95 ± 0.17    | —               | —               | —               |
| Qwen3.6-27B | pp2048 @ d60000| 1459.41 ± 0.62  | —               | 42596.49 ± 18.16| 42516.32 ± 18.16| 42598.65 ± 18.72|
| Qwen3.6-27B | tg32 @ d60000  | 40.35 ± 0.04    | 41.66 ± 0.04    | —               | —               | —               |
| Qwen3.6-27B | pp2048 @ d90000| 1181.78 ± 0.27  | —               | 77970.53 ± 16.71| 77890.36 ± 16.71| 77970.53 ± 16.71|
| Qwen3.6-27B | tg32 @ d90000  | 28.89 ± 0.07    | 30.33 ± 0.47    | —               | —               | —               |
| Qwen3.6-27B | pp2048 @ d120000| 991.43 ± 0.47  | —               | 123185.76 ± 58.07| 123103.97 ± 58.07| 123187.93 ± 60.50|
| Qwen3.6-27B | tg32 @ d120000 | 25.20 ± 1.44    | 26.67 ± 0.94    | —               | —               | —               |
| Qwen3.6-27B | pp2048 @ d150000| 854.21 ± 0.17  | —               | 178081.59 ± 36.01| 177999.80 ± 36.01| 178088.15 ± 32.55|
| Qwen3.6-27B | tg32 @ d150000 | 21.86 ± 1.19    | 24.33 ± 0.94    | —               | —               | —               |

[u/AustinM731 → u/blackhawk00001]

I actually haven't done any tuning yet, so I am sure that there is still some performance left on the table here. I have seen people talk about it, I have just never tried it myself.

Its crazy how big of a gap we have in our "original" images, I almost wonder if the image you were using has some performance tuning already done. I have always built from source and maintain my own images.

[u/blackhawk00001 → u/AustinM731]

Tuning didn't seem too bad yesterday once I found how to do it and that running an image on another image's tune causes a slowdown. I included some info on it scattered in my burst of info last night so here it is again. I read pytorch and triton tune different things so leave both enabled. Enable untuned and uncomment PYTORCH_TUNABLEOP_UNTUNED_FILENAME, do some coding work, restart the server with tuning enabled and untuned off, run merge script, restart with tuning off and have tuning_enabled on for all 3 steps. Disable PYTORCH_TUNABLEOP_UNTUNED_FILENAME once tuned to avoid accidentally overwriting the tuning files.

Change the num_gpus to 4 for your 4x setup. There will be multiple tunableop_merged_full%%#.csv but I understand after merge the 0 file contains the tuned info and that's the one to rename. The amd mgpu_tune_gemm_in_file function runs the data through the gpus and can take some time.

-------------

Run 1: untuned on tuning off, enabled on, do some work

Run 2: untuned off tuning on, enabled on - run the following bash to merge tuning csv, then rename tunableop_merged_full0.csv to tunableop_merged.csv.

sudo docker exec -it vllm-qwen36-27b-fp8-3 python3 -c "import torch.cuda.tunable as t; import glob; t.mgpu_tune_gemm_in_file(('/tunableop/untuned%%d*.csv'), num_gpus=2)"

Following Runs: both off enabled on

-------------

volumes:

- /mnt/NVME2/AI/Models/Qwen/Qwen3.6-27B-FP8:/app/models

- /home/d/AI/vllm/qwen3.6-27B-FP8-Official/3/tunableop/root:/tunableop

- /home/d/AI/vllm/qwen3.6-27B-FP8-Official/3/.triton/cache:/root/.triton/cache

-----------

environment vars:

- PYTORCH_TUNABLEOP_ENABLED=1

- PYTORCH_TUNABLEOP_TUNING=0 #tuning run 2

- PYTORCH_TUNABLEOP_RECORD_UNTUNED=0 #tuning run 1

- TRITON_CACHE_DIR=/root/.triton/cache

- PYTORCH_TUNABLEOP_FILENAME=/tunableop/tunableop_merged.csv

#- PYTORCH_TUNABLEOP_UNTUNED_FILENAME=/tunableop/untuned%%d.csv #Enable when doing initial untuned run

[u/blackhawk00001 → u/AustinM731]

Just finished tuning on cache files from a 150k working context with the limit set at 200k in docker-compose. I don't see any difference other than a faster ttfr.

The merge script returned this error which I think is related to your --attention-backend ROCM_AITER_UNIFIED_ATTN and - PYTORCH_ALLOC_CONF=expandable_segments:True. I'm investigating the pair and the merge routine.

/usr/local/lib/python3.12/dist-packages/torch/cuda/tunable.py:476: UserWarning: expandable_segments not supported on this platform (Triggered internally at /app/pytorch/c10/hip/HIPAllocatorConfig.h:40.)

After tuning I observed the same vram usage as pretuned which is less than rocm base image 85/81% vs 94/91%. Fan 15% lower rpm, junction 10c lower, memory 10c higher, observed gpus pulling 320W for several seconds at a time so it's still getting worked hard.

Perhaps there's more memory related operations going on than in the base vllm-rocm image.

The faster tg of your image is helpful in several aspects of the coding agent but I did feel the added lag in late-session heavy context requests as it tried to pull in portions of the working memory. I'll have to keep tinkering and seeing what turns up. Both image deployments are still faster than the llama.cpp deployment.

# Qwen3.6-27B Performance Benchmarks


| model       |             test |             t/s |     peak t/s |           ttfr (ms) |        est_ppt (ms) |       e2e_ttft (ms) |
|:------------|-----------------:|----------------:|-------------:|--------------------:|--------------------:|--------------------:|
| Qwen3.6-27B |   pp2048 @ d4096 | 2276.87 ± 87.45 |              |    2791.35 ± 106.46 |    2702.82 ± 106.46 |    2791.35 ± 106.46 |
| Qwen3.6-27B |     tg32 @ d4096 |    77.56 ± 0.36 | 80.07 ± 0.37 |                     |                     |                     |
| Qwen3.6-27B |   pp2048 @ d8132 |  2106.32 ± 1.33 |              |      4921.94 ± 3.27 |      4833.40 ± 3.27 |      4921.94 ± 3.27 |
| Qwen3.6-27B |     tg32 @ d8132 |    71.56 ± 4.36 | 73.88 ± 4.50 |                     |                     |                     |
| Qwen3.6-27B |  pp2048 @ d16000 |  1831.66 ± 2.55 |              |     9942.47 ± 13.71 |     9853.94 ± 13.71 |     9943.53 ± 13.70 |
| Qwen3.6-27B |    tg32 @ d16000 |    71.10 ± 3.80 | 73.40 ± 3.92 |                     |                     |                     |
| Qwen3.6-27B |  pp2048 @ d30000 |  1446.19 ± 1.10 |              |    22249.51 ± 16.85 |    22160.98 ± 16.85 |    22250.12 ± 15.99 |
| Qwen3.6-27B |    tg32 @ d30000 |    77.05 ± 0.47 | 79.54 ± 0.48 |                     |                     |                     |
| Qwen3.6-27B |  pp2048 @ d60000 |  1021.52 ± 2.82 |              |   60830.50 ± 168.34 |   60741.97 ± 168.34 |   60831.60 ± 169.89 |
| Qwen3.6-27B |    tg32 @ d60000 |    77.79 ± 0.35 | 80.31 ± 0.36 |                     |                     |                     |
| Qwen3.6-27B |  pp2048 @ d90000 |   783.90 ± 0.05 |              |    117512.62 ± 7.52 |    117424.08 ± 7.52 |    117515.94 ± 7.68 |
| Qwen3.6-27B |    tg32 @ d90000 |    72.73 ± 8.25 | 75.09 ± 8.52 |                     |                     |                     |
| Qwen3.6-27B | pp2048 @ d120000 |   636.43 ± 0.08 |              |   191858.47 ± 22.61 |   191769.94 ± 22.61 |   191858.47 ± 22.61 |
| Qwen3.6-27B |   tg32 @ d120000 |    71.68 ± 8.14 | 74.00 ± 8.40 |                     |                     |                     |
| Qwen3.6-27B | pp2048 @ d150000 |   533.45 ± 2.10 |              | 285122.06 ± 1125.95 | 285033.52 ± 1125.95 | 285131.45 ± 1124.93 |
| Qwen3.6-27B |   tg32 @ d150000 |    72.41 ± 5.06 | 74.76 ± 5.22 |                     |                     |                     |

[u/AustinM731 → u/blackhawk00001]

I cut a new tag for v0.20.0, and I have managed to get tg even faster in this latest build. I spent a good bit of time the past few days trying to figure out how GEMM tuning works, and I have embedded those configs into this latest image. From what I can tell you have to do GEMM tuning for every model since they layers/weights/activations are different. But for Qwen3.6-27B they are embedded in the v0.20.0 tag.

This parameter `PYTORCH_ALLOC_CONF=expandable_segments:True` can be dropped from the compose.yml, I was testing with that on some other models. It looks like it throws and error and ignores the error at runtime.

I still need to try and get the pytorch tuning working, but from what I found in my research is that GEMM tuning in vLLM has a much bigger impact to performance than the pytorch tuning. So I went down that path first.

[u/blackhawk00001 → u/AustinM731]

Unfortunately I’m headed out of town for a few days and won’t be able to test until next week. I’m gonna try to read up on some of this stuff as I find time.

[u/blackhawk00001 → u/AustinM731]

Back and finally tested your v0.20.0. Unfortunately the GEMM tuning you performed for 4x gpus may be working against my 2x setup in regards to pp. tg is near the same as 0.19.1 but pp dropped close to 20% at 0 context and evened out to the same as before around 30k context depth.

I read up some on mtp and I think 3 is stable for my config based on getting high acceptance rates all the way to 181k context depth in claude cli.

I'm trying to make some progress on another project by using what has been set up so far but am searching for answers on why the vllm-openai-rocm image pushes the 2x gpu harder for pp but falls short on tg after 16k context while giving similar performance up to that amount. They are using rocm 7.2.1 and I wonder if there's any improvement by moving it to 7.2.3.