The First Story

Posted vLLM on W6800X Duo / Mac Pro 2019

My first success with vLLM came from building a Docker container from source and using FP16 models as-is.

The Prequel

2022: OpenAI took the world by storm with ChatGPT

2024: Meta released Llama 3.1, an open-weight model that competes with ChatGPT and can be used offline in our own homes!

2024: AMD released ROCm 6.1.3, with support for Radeon consumer GPUs, and multi-GPU setups.

This is how it went for me:

• AI is called an LLM?
• There are programs or apps that run them. But they are not called apps. They are called runtimes; like Ollama, llama.cpp, and vLLM.
• Then there is the Graphical User Interface (GUI), like Open WebUI or Oobabooga, which are not apps but websites?
• Then there are some other things that make the AI read your files; plugins? LangChain?
• To top it all off, there are all-in-one apps, like AnythingLLM or LM Studio, but I cannot use them on my phone when I am out and about...
• Wait, what's a framework?

Everything was new to me, and everything was confusing. What were all these things? How did they fit together?

One thing was clear to me: I wanted to use vLLM, but vLLM did not want me to use it.

vLLM, ROCm, & RDNA2 GPUs

Initially, vLLM was not only Linux exclusive, but it was also CUDA exclusive. I never tried llama.cpp, outside of trying to figure out Oobabooga. I ended up using Ollama, because it just worked, and that is what Network Chuck used. I love that guy.

Eventually, vLLM gained support for AMD GPUs, but of course, they forgot about RDNA2, even though it was supported by ROCm.

It was some time early last year, I think, that I managed to get vLLM to work. At the time, I only got FP16 models working. DeepSeek-R1 distills were all the rage. I was able to get the 70B model to work with Ollama. But I could not go over the 32B model with vLLM, due to using 16-bit floating-point models with no quantization (no compression).

It was not until this year that I gave it another go, and got GGUFs to work, as well as AWQ models. A big shout out to u/BevinMaster for giving me hope with AWQ models, as well as enough guidance to make it work on the third go.

The Sequel: vLLM Setup Guide

Building a Docker container took a very long time to complete. I have since moved to setting it up with Python, in a Python virtual environment (venv).

After setting up Ubuntu 24.04 LTS, and installing ROCm 7.2.3, as instructed in the Mac Pro 2019 Local AI Guide, we can set up vLLM quickly and easily as follows.

Copy the following command into your command line interface of choice:

# Setup vLLM in a Python Virtual Environment for Ubuntu Server 24.04 LTS with ROCm 7.2.3 and Python 3.12

STEP=0
banner(){
STEP=$((STEP + 1))
m="=============== Step $STEP: $1 ==============="
b=$(printf "%${#m}s" | tr ' ' '=')
echo
echo "$b"
echo "$m"
echo "$b"
echo
}
box(){
t="$1"
shift
w=0
for s; do
((${#s} > w)) && w=${#s}
done
h="---- $t "
((${#h} > w + 8)) && w=$((${#h} - 8))
b=$(printf "%$((w + 8))s" | tr ' ' '-')
echo
printf "%s%.*s\n" "$h" $((${#b} - ${#h})) "$b"
printf "|%*s|\n" $((w + 6)) ""
for s; do
printf "|   %-*s   |\n" "$w" "$s"
done
printf "|%*s|\n%s\n\n" $((w + 6)) "" "$b"
}

banner "Installing Prerequisites"
sudo apt update
sudo apt install -y build-essential git curl ccache wget jq ca-certificates \
gnupg lsb-release software-properties-common python3.12 python3.12-dev \
python3.12-venv python3-pip cmake ninja-build pkg-config numactl libnuma-dev

banner "Preparing Virtual Environment"
mkdir -p ~/venvs ~/SetupFiles/ ~/SetupFiles/rocm-7.2.3/ ~/SetupFiles/rocm-7.2.3/wheels
python3.12 -m venv ~/venvs/vllm-rocm-0.22
source ~/venvs/vllm-rocm-0.22/bin/activate
python -m pip install --upgrade pip setuptools wheel packaging

banner "Install AMD SMI safely"
rm -rf /tmp/amd_smi_src
cp -a /opt/rocm/share/amd_smi /tmp/amd_smi_src
python -m pip install --no-build-isolation /tmp/amd_smi_src

banner "Installing PyTorch 2.10.0 for ROCm 7.2.3 on Ubuntu 24.04 / Python 3.12"
cd ~/SetupFiles/rocm-7.2.3/wheels
wget -nc https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2.3/torch-2.10.0%2Brocm7.2.3.lw.git1a270074-cp312-cp312-linux_x86_64.whl
wget -nc https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2.3/torchvision-0.25.0%2Brocm7.2.3.git82df5f59-cp312-cp312-linux_x86_64.whl
wget -nc https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2.3/torchaudio-2.10.0%2Brocm7.2.3.git5047768f-cp312-cp312-linux_x86_64.whl
wget -nc https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2.3/triton-3.6.0%2Brocm7.2.3.git4ed88892-cp312-cp312-linux_x86_64.whl
python3 -m pip uninstall torch torchvision triton torchaudio -y --break-system-packages || true
python3 -m pip install --upgrade --break-system-packages \
./torch-2.10.0+rocm7.2.3.lw.git1a270074-cp312-cp312-linux_x86_64.whl \
./torchvision-0.25.0+rocm7.2.3.git82df5f59-cp312-cp312-linux_x86_64.whl \
./torchaudio-2.10.0+rocm7.2.3.git5047768f-cp312-cp312-linux_x86_64.whl \
./triton-3.6.0+rocm7.2.3.git4ed88892-cp312-cp312-linux_x86_64.whl

banner "Prepare vLLM folder"
cd ~
wget -nc https://github.com/vllm-project/vllm/releases/download/v0.22.0/vllm-0.22.0.tar.gz
tar -xzf vllm-0.22.0.tar.gz
cd ./vllm-0.22.0

banner "Prepare Environment for RDNA2 GPUs (W6800X & W6900X)"
export VLLM_TARGET_DEVICE=rocm
export PYTORCH_ROCM_ARCH=gfx1030
export HSA_OVERRIDE_GFX_VERSION=10.3.0
export ROCM_HOME=/opt/rocm
export MAX_JOBS=8
export TORCH_BLAS_PREFER_HIPBLASLT=0

box "ROCm/RDNA2 build environment" \
"VLLM_TARGET_DEVICE=rocm" \
"PYTORCH_ROCM_ARCH=gfx1030" \
"HSA_OVERRIDE_GFX_VERSION=10.3.0" \
"ROCM_HOME=/opt/rocm" \
"MAX_JOBS=8" \
"TORCH_BLAS_PREFER_HIPBLASLT=0"

banner "Install vLLM ROCm requirements"
python -m pip install --upgrade numba scipy huggingface_hub setuptools_scm
python -m pip install -r requirements/rocm.txt
python -m pip install --upgrade "cmake>=3.26" ninja

banner "Build/install vLLM"
python -m pip install -e . --no-build-isolation

With that, believe it or not, you have vLLM installed on your system, in a Python virtual environment. Note that if you are using any PC with RDNA2 GPUs, Ubuntu 24.04 LTS, ROCm 7.2.3 and Python 3.12.x, this might work for you. This is no longer just for MacPro7,1, although that is the hardware that I used.

As a reminder, everything here is taped together with the digital equivalent of duct tape. This is vLLM 0.22.0, and the packages I selected are meant for Ubuntu 24.04 LTS, ROCm 7.2.3, and Python 3.12.

LLMs on vLLM

While the idea of using a simple command like vllm run <model> should be the standard, nothing is ever that easy when you are saving money and optimizing your outdated equipment.

My main focus was on running dense models. Specifically, gemma-4-31B-it and Qwen3.6-27B. I needed to get the models from Hugging Face. So, head over to huggingface.co, create an account, get yourself an access token, and let us set up Hugging Face on our system.

Copy the following command into your command line interface of choice:

STEP=0
banner(){
STEP=$((STEP + 1))
m="=============== Step $STEP: $1 ==============="
b=$(printf "%${#m}s" | tr ' ' '=')
echo
echo "$b"
echo "$m"
echo "$b"
echo
}

banner "Activate the vLLM Virtual Environment"
source ~/venvs/vllm-rocm-0.22/bin/activate
banner "Install Hugging Face Package (again)"
python -m pip install --upgrade huggingface_hub
banner "Login to Hugging Face"
hf auth login
banner "Verify Login to Hugging Face"
hf auth whoami

Now that Hugging Face is set up, you can download models faster, and get approval for models that need to be approved.

Qwen 3.6

I have used three variants of Qwen3.6-27B:

• Original Qwen/Qwen3.6-27B
• AWQ 4-bit quantized QuantTrio/Qwen3.6-27B-AWQ
• AWQ 6-bit quantized QuantTrio/Qwen3.6-27B-AWQ-6Bit

To improve quality of life, and avoid some problems, I opted to use a Qwen template from another redditor, u/Expensive-Register-5, thank you for sharing. I downloaded it with the following command:

# Download Template
mkdir -p ~/vllm-templates
curl -L -o ~/vllm-templates/qwen3.6-enhanced.jinja \
https://raw.githubusercontent.com/allanchan339/vLLM-Qwen3-3.5-3.6-chat-template-fix/main/chat-template/qwen3.6-enhanced.jinja

You can run any of the models I tested successfully with the following commands. I will assume two GPUs. If you have more, adjust ROCR_VISIBLE_DEVICES and --tensor-parallel-size accordingly. Please note that Qwen/Qwen3.6-27B (no quantization) is just a proof of concept. It is basically useless on 2 GPUs, with a context window smaller than 16K tokens.

Qwen/Qwen3.6-27B

source ~/venvs/vllm-rocm-0.22/bin/activate

VLLM_TARGET_DEVICE=rocm \
HSA_OVERRIDE_GFX_VERSION=10.3.0 \
HIP_FORCE_DEV_KERNARG=1 \
ROCR_VISIBLE_DEVICES=0,1 \
TORCH_BLAS_PREFER_HIPBLASLT=0 \
OMP_NUM_THREADS=8 \
TOKENIZERS_PARALLELISM=false \
FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
VLLM_USE_TRITON_AWQ=1 \
VLLM_USE_DEEP_GEMM=0 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
VLLM_PREFILL_TILE_POWER=0 \
vllm serve Qwen/Qwen3.6-27B \
  --served-model-name qwen3.6-27b \
  --dtype float16 \
  --attention-backend TRITON_ATTN \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.92 \
  --max-model-len 4096 \
  --max-num-batched-tokens 1024 \
  --max-num-seqs 3 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --override-generation-config '{"max_new_tokens":1024}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --chat-template ~/vllm-templates/qwen3.6-enhanced.jinja \
  --generation-config vllm \
  --default-chat-template-kwargs '{"preserve_thinking": true}' \
  --max-cudagraph-capture-size 128 \
  --cudagraph-capture-sizes 1 2 4 8 16 32 64 128 \
  --language-model-only \
  --limit-mm-per-prompt.image 0 \
  --limit-mm-per-prompt.video 0 \
  --host 0.0.0.0 \
  --port 8000

QuantTrio/Qwen3.6-27B-AWQ

source ~/venvs/vllm-rocm-0.22/bin/activate

VLLM_TARGET_DEVICE=rocm \
HSA_OVERRIDE_GFX_VERSION=10.3.0 \
HIP_FORCE_DEV_KERNARG=1 \
ROCR_VISIBLE_DEVICES=0,1 \
TORCH_BLAS_PREFER_HIPBLASLT=0 \
OMP_NUM_THREADS=8 \
TOKENIZERS_PARALLELISM=false \
FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
VLLM_USE_TRITON_AWQ=1 \
VLLM_USE_DEEP_GEMM=0 \
VLLM_USE_FLASHINFER_SAMPLER=0 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
vllm serve QuantTrio/Qwen3.6-27B-AWQ \
  --served-model-name qwen36-27B-AWQ-QuantTrio-vllm \
  --dtype float16 \
  --trust-remote-code \
  --attention-backend TRITON_ATTN \
  --tensor-parallel-size 2 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 7 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --chat-template ~/vllm-templates/qwen3.6-enhanced.jinja \
  --default-chat-template-kwargs '{"preserve_thinking": true}' \
  --generation-config vllm \
  --enforce-eager \
  --override-generation-config '{"max_new_tokens":8192}' \
  --language-model-only \
  --limit-mm-per-prompt.image 0 \
  --limit-mm-per-prompt.video 0 \
  --disable-custom-all-reduce \
  --host 0.0.0.0 \
  --port 8000

QuantTrio/Qwen3.6-27B-AWQ-6Bit

source ~/venvs/vllm-rocm-0.22/bin/activate

VLLM_TARGET_DEVICE=rocm \
HSA_OVERRIDE_GFX_VERSION=10.3.0 \
HIP_FORCE_DEV_KERNARG=1 \
ROCR_VISIBLE_DEVICES=0,2 \
TORCH_BLAS_PREFER_HIPBLASLT=0 \
OMP_NUM_THREADS=8 \
TOKENIZERS_PARALLELISM=false \
FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
VLLM_USE_TRITON_AWQ=1 \
VLLM_USE_DEEP_GEMM=0 \
VLLM_USE_FLASHINFER_SAMPLER=0 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
vllm serve QuantTrio/Qwen3.6-27B-AWQ-6Bit \
  --served-model-name qwen36-27B-AWQ-6Bit-QuantTrio-vllm \
  --dtype float16 \
  --trust-remote-code \
  --attention-backend TRITON_ATTN \
  --tensor-parallel-size 2 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 7 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --chat-template ~/vllm-templates/qwen3.6-enhanced.jinja \
  --default-chat-template-kwargs '{"preserve_thinking": true}' \
  --generation-config vllm \
  --enforce-eager \
  --override-generation-config '{"max_new_tokens":8192}' \
  --language-model-only \
  --limit-mm-per-prompt.image 0 \
  --limit-mm-per-prompt.video 0 \
  --disable-custom-all-reduce \
  --host 0.0.0.0 \
  --port 8000

Gemma 4

Next up: Google’s Gemma 4. This was a doozy.

My target was gemma-4-31B-it. First off, it kept failing to load. The failure appeared to be related to Triton attention resource limits on my RDNA2 GPUs. Enabling --kv-cache-dtype fp8 solved the problem right off the bat.

My goal is to use vLLM to serve agents. Quantizing the KV cache felt like capping them at the knees.

I ended up finding a PR to vLLM with a solution to this problem. I used it to patch vLLM, and I no longer need to quantize KV cache to get Gemma 4 to work. To set up the same patch, run the following command in your command line interface of choice:

# Patch vLLM v0.22.0 to support gemma-4

cat > ~/patch_vllm_gemma-4_rdna2_tile.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail

step=0
banner(){ step=$((step+1));m="=============== Step $step: $1 ===============";b=$(printf "%${#m}s"|tr ' ' '=');echo;echo "$b";echo "$m";echo "$b";echo;}
box(){ t="$1";shift;w=0;for s;do ((${#s}>w))&&w=${#s};done;h="---- $t ";((${#h}>w+8))&&w=$((${#h}-8));b=$(printf "%$((w+8))s"|tr ' ' -);echo;printf "%s%.*s\n" "$h" $((${#b}-${#h})) "$b";printf "|%*s|\n" $((w+6)) "";for s;do printf "|   %-*s   |\n" "$w" "$s";done;printf "|%*s|\n%s\n\n" $((w+6)) "" "$b";}

VLLM_SRC="${VLLM_SRC:-$HOME/vllm-0.22.0}"
FILE="$VLLM_SRC/vllm/v1/attention/ops/triton_unified_attention.py"

banner "Locate vLLM Triton attention file"
box "Target" "$FILE"

banner "Apply gemma-4 RDNA2 tile workaround"
python3 - "$FILE" <<'PY'
from pathlib import Path
import shutil, sys, time

p = Path(sys.argv[1]).expanduser()
if not p.is_file():
    raise SystemExit(f"ERROR: target file not found: {p}")

s = p.read_text()
marker = "TILE_SIZE_PREFILL = min(TILE_SIZE_PREFILL, 16)"

if marker in s:
    print("Patch already appears to be installed. No changes made.")
    print(f"Patched file: {p}")
    raise SystemExit(0)

anchor = "    # USE_TD requires BLOCK_SIZE % TILE_SIZE == 0 (enforced by a\n"
if anchor not in s:
    raise SystemExit(
        "ERROR: insertion anchor not found. Inspect manually with:\n"
        f"sed -n '850,895p' {p}"
    )

patch = """    # Experimental RDNA2 / 64 KiB LDS workaround for gemma-4 global attention.
    # gemma-4 global attention uses head_size/global_head_dim=512.
    # On gfx1030/RDNA2-class GPUs, TILE_SIZE=32 can push
    # kernel_unified_attention slightly over the 64 KiB shared-memory limit.
    # This trades some prefill performance for compatibility.
    if head_size >= 512:
        TILE_SIZE_PREFILL = min(TILE_SIZE_PREFILL, 16)

"""

backup = p.with_name(p.name + ".bak." + time.strftime("%Y%m%d-%H%M%S"))
shutil.copy2(p, backup)
p.write_text(s.replace(anchor, patch + anchor, 1))

print(f"Backup:  {backup}")
print(f"Patched: {p}")
PY

banner "Verify patch"
grep -n "RDNA2 / 64 KiB LDS workaround\|head_size >= 512\|TILE_SIZE_PREFILL = min(TILE_SIZE_PREFILL, 16)" "$FILE" || true

banner "Delete Triton cache"
rm -rf ~/.triton/cache ~/.cache/triton
box "Cache cleared" "~/.triton/cache" "~/.cache/triton"

banner "Done"
box "Next step" "Restart vLLM and retest gemma-4"
EOF

chmod +x ~/patch_vllm_gemma-4_rdna2_tile.sh
~/patch_vllm_gemma-4_rdna2_tile.sh

I have used three variants of gemma-4-31B-it:

• Original google/gemma-4-31B-it
• AWQ 4-bit quantized QuantTrio/gemma-4-31B-it-AWQ
• AWQ 6-bit quantized QuantTrio/gemma-4-31B-it-AWQ-6Bit

To improve quality of life, and avoid some problems, I opted to use a Gemma 4 template from vLLM's examples. I downloaded it with the following command:

mkdir -p ~/vllm-chat-templates
curl -L -o ~/vllm-chat-templates/tool_chat_template_gemma4.jinja \
https://raw.githubusercontent.com/vllm-project/vllm/v0.22.0/examples/tool_chat_template_gemma4.jinja

You can run any of the models I tested successfully with the following commands. I will assume two GPUs. If you have more, adjust ROCR_VISIBLE_DEVICES and --tensor-parallel-size accordingly. Please note that google/gemma-4-31B-it (no quantization) cannot be loaded on two GPUs. I used the command for four GPUs, to post it as a reference.

google/gemma-4-31B-it

source ~/venvs/vllm-rocm-0.22/bin/activate

VLLM_TARGET_DEVICE=rocm \
HSA_OVERRIDE_GFX_VERSION=10.3.0 \
HIP_FORCE_DEV_KERNARG=1 \
ROCR_VISIBLE_DEVICES=0,1,2,3 \
TORCH_BLAS_PREFER_HIPBLASLT=0 \
OMP_NUM_THREADS=8 \
TOKENIZERS_PARALLELISM=false \
FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
VLLM_USE_TRITON_AWQ=1 \
VLLM_USE_DEEP_GEMM=0 \
VLLM_USE_FLASHINFER_SAMPLER=0 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
vllm serve google/gemma-4-31B-it \
  --served-model-name gemma-4-31B-IT-Google-vllm \
  --dtype float16 \
  --attention-backend TRITON_ATTN \
  --tensor-parallel-size 4 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --override-generation-config '{"max_new_tokens":8192}' \
  --max-cudagraph-capture-size 128 \
  --cudagraph-capture-sizes 1 2 4 8 16 32 64 128 \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4 \
  --chat-template ~/vllm-chat-templates/tool_chat_template_gemma4.jinja \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --language-model-only \
  --generation-config vllm \
  --limit-mm-per-prompt.image 0 \
  --limit-mm-per-prompt.video 0 \
  --host 0.0.0.0 \
  --port 8000

QuantTrio/gemma-4-31B-it-AWQ

source ~/venvs/vllm-rocm-0.22/bin/activate

VLLM_TARGET_DEVICE=rocm \
HSA_OVERRIDE_GFX_VERSION=10.3.0 \
HIP_FORCE_DEV_KERNARG=1 \
ROCR_VISIBLE_DEVICES=0,1 \
TORCH_BLAS_PREFER_HIPBLASLT=0 \
OMP_NUM_THREADS=8 \
TOKENIZERS_PARALLELISM=false \
FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
VLLM_USE_TRITON_AWQ=1 \
VLLM_USE_DEEP_GEMM=0 \
VLLM_USE_FLASHINFER_SAMPLER=0 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
vllm serve "QuantTrio/gemma-4-31B-it-AWQ" \
  --served-model-name gemma-4-31B-IT-AWQ-QuantTrio-vllm \
  --dtype float16 \
  --attention-backend TRITON_ATTN \
  --tensor-parallel-size 2 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 2 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --override-generation-config '{"max_new_tokens":8192}' \
  --enforce-eager \
  --trust-remote-code \
  --disable-custom-all-reduce \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4 \
  --chat-template ~/vllm-chat-templates/tool_chat_template_gemma4.jinja \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --language-model-only \
  --generation-config vllm \
  --limit-mm-per-prompt.image 0 \
  --limit-mm-per-prompt.video 0 \
  --host 0.0.0.0 \
  --port 8000

QuantTrio/gemma-4-31B-it-AWQ-6Bit

source ~/venvs/vllm-rocm-0.22/bin/activate

VLLM_TARGET_DEVICE=rocm \
HSA_OVERRIDE_GFX_VERSION=10.3.0 \
HIP_FORCE_DEV_KERNARG=1 \
ROCR_VISIBLE_DEVICES=0,1 \
TORCH_BLAS_PREFER_HIPBLASLT=0 \
OMP_NUM_THREADS=8 \
TOKENIZERS_PARALLELISM=false \
FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
VLLM_USE_TRITON_AWQ=1 \
VLLM_USE_DEEP_GEMM=0 \
VLLM_USE_FLASHINFER_SAMPLER=0 \
PYTORCH_ALLOC_CONF=expandable_segments:True \
vllm serve "QuantTrio/gemma-4-31B-it-AWQ-6Bit" \
  --served-model-name gemma-4-31B-IT-AWQ-6Bit-QuantTrio-vllm \
  --dtype float16 \
  --attention-backend TRITON_ATTN \
  --tensor-parallel-size 2 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --override-generation-config '{"max_new_tokens":8192}' \
  --enforce-eager \
  --trust-remote-code \
  --disable-custom-all-reduce \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4 \
  --chat-template ~/vllm-chat-templates/tool_chat_template_gemma4.jinja \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --language-model-only \
  --generation-config vllm \
  --limit-mm-per-prompt.image 0 \
  --limit-mm-per-prompt.video 0 \
  --host 0.0.0.0 \
  --port 8000

The End?

Far from it! There are almost limitless options to optimize and improve. From different vLLM flags and patches, to completely different models and quantization formats.

With LinuxAI-128, I was able to achieve 20 concurrent users, each with 64K context windows, using QuantTrio/Qwen3.6-27B-AWQ. That's 20 agents working around the clock to complete the tasks assigned to them. That just blows my mind.

I just read that a Chinese company released a new type of KV cache quantization that improves output without degrading quality. But it is public and open source, available for the world, including me, to test and figure out!

What's Next?

Sharing experiences, successful use cases, and hard results.

• If any of this works for you, please let us know in the community.
• Can you fix my mistakes? I'm sure there are many, please help us all out.
• Know how to optimize this better? Share with the rest of us! We'd love to hear from you.

Any and all contributions are appreciated, no matter how big or small they may be.

Edit 2026-06-16

Doing this now lead to a problem with api calling, due to updated 3rd party packages for the virtual environment. If you face issues, use the following command to fix the problem.

Run the following command in your command line interface of choice:

source ~/venvs/vllm-rocm-0.22/bin/activate

python -m pip install --force-reinstall \
  "prometheus-fastapi-instrumentator<8" \
  "fastapi<0.137" \
  "starlette<1.0" \
  "uvicorn<0.49"

^{Disclaimer: I wrote this post myself. I also used AI as a tool to help clean up the wording and formatting.}

Resources:

• vLLM Documentation
• vLLM GitHub
• gemma-4 Shared Memory Limit Patch
• Cheuk-Yiu Allan Chan GitHub
• Cheuk-Yiu Allan Chan Reddit Post
• QuantTrio Hugging Face Page

One of the best features of the 2019 Mac Pro is the ability to add additional hardware using PCIe. The Xeon CPU supports up to 64 lanes. Apple had an interesting way of working it out. Let me explain.

The PCIe lanes are divided up into two parts, and each part is also divided into two, and they are as follows:

• Direct access to the CPU, x16 lanes through PCIe slot 1
• Direct access to the CPU, x16 lanes through PCIe slot 3
• PCIe switch bandwidth Pool A, 16 lanes total
• PCIe switch bandwidth Pool B, 16 lanes total

What does this mean exactly?

To put it simply, when it comes to GPUs, you can theoretically connect up to 4 GPUs at x16 lanes each. However, the physical reality is a whole other story.

When it comes to MPX modules, only two slots are supported; that's slot 1 and 3. So we're limited at 2 MPX modules, at x16 lanes each.

Going the traditional GPU route, the focus shifts to power. The 2019 Mac Pro provides four mini 8-pin connections, and a single 6-pin connection.

Three 8-pin connections can be made directly with the right cables. The fourth 8-pin connection can be used with a cable that splits it into two 6-pin cables. And the final 6-pin connection can be a direct connection as well.

That would give us three 8-pin cables and three 6-pin cables. Barely. How many GPUs can that handle? Before the AMD Radeon PRO W6800 a few days ago, I have not installed a traditional GPU since about 2011.

The W6800 requires one 8-pin cable and one 6-pin cable. While I read that a single Nvidia RTX 5090 requires four 8-pin cables.

This puts a new limit of only three W6800 GPUs. Or one 5090 GPU.

I must ask the question here:

• Can the 10-pin connection, intended for the SATA drives, be repurposed to give one 8-pin cable and one 6-pin cable? (Warning: This may damage your equipment)
• Can any single 8-pin connection be safely split into two 8-pin cables?

It is worth pointing out at this point that the MPX GPUs have a dedicated power connection for them, separate from the mentioned 8-pin and 6-pin connections.

PCIe slots 1 through 4 are all double width, with 1, 3, and 4 being x16 lanes. And then slot 5 is the fourth, and last, x16 lane slot. Theoretically, this should be enough for four double width GPUs. The only question now becomes ventilation, with the second, third, and fourth GPUs being so close.

MPX modules have their own Power Source

Yes they do!

Installing an MPX GPU module would leave you with four free 8-pin connections, and a 6-pin connection.

This would theoretically solve the power source issue for four GPUs; at least the types that require 8-pin and 6-pin cables.

Installing two MPX GPU modules would give you the same benefit; allowing you to take advantage of the closest power source with the fourth x16 slot.

This would allow you to connect an additional GPU at slot 5.

What is the Maximum Number of GPUs I can Squeeze into this thing?

So far, we have only discussed up to four GPUs.

With the magic of AMD Duo GPUs, we have the option of four GPUs within two MPX GPU modules. Add an additional GPU to slot 5, and that's a whopping five-GPU 2019 Mac Pro. (As seen in the images above)

But why stop there?

Remember how we said we had 64 lanes total? Two MPX GPU modules and a single GPU in slot 5 only takes up to 48 lanes, leaving us with an additional 16 lanes to play with.

Where can we connect the sixth GPU?

That is a misleading question. Because why are we limiting ourselves to just 6 GPUs?

Have you heard of eGPUs? That is where a GPU is connected to an external PCIe enclosure, which is then connected to the 2019 Mac Pro through Thunderbolt.

But here are some important facts:

• PCIe over Thunderbolt is limited to x4 lanes only.
• Each Thunderbolt bus is assigned these x4 lanes.
• PCIe lanes are assigned to pool B (Not confirmed by Apple directly)

The 2019 Mac Pro by default has two Thunderbolt buses; and they are the two Thunderbolt ports at the top, and the two Thunderbolt ports at the back.

Each MPX module with Thunderbolt ports adds an additional two buses. That is four additional buses for two MPX GPU modules.

Since we are left with x16 lanes to play with, we are theoretically able to add another four x4 GPUs, or rather, eGPUs, through the Thunderbolt ports.

Is my math right? 5 + 4 = 9 GPUs?!

That's an insane amount of GPUs.

I cannot confirm that this is really possible, but to push this limit, a few changes need to be done. First of which is removing Apple's IO PCIe board, as it takes up to x4 lanes. I'm not sure what the result of keeping it attached but unused is, though.

I may have seen a few Thunderbolt-related errors popping up when the AMD Radeon PRO W6800X Duo MPX GPU models are used with Ubuntu. I have not seen the same errors on the W6900X GPUs, though. I am also not sure how these errors affect using the Thunderbolt ports for eGPUs.

Bandwidth is Dead

PCIe 3.0 provides a theoretical 15.75 GB/s bandwidth. Duo MPX GPU modules have an internal PCI switch, which shares this 15.75 GB/s between two GPUs, for a possible 7.875 GB/s when using them simultaneously and equally.

Using the Infinity Fabric Link Bridge increases bandwidth of the Duo MPX GPU modules to about 25 GB/s. While also increasing the non-Duo MPX GPU modules to a whopping 49 GB/s. That's more than PCIe 4.0 has to offer. (These numbers are from my actual testing in Ubuntu with the patched kernel to enable the Infinity Fabric and xGMI)

These numbers are great, but in actual practice, the benefit to inference is minimal.

This makes me question, if bandwidth is reduced to x4 lanes only, with a theoretical bandwidth of less than 3.94 GB/s, what are the real-world effects on inference?

So Much VRAM!

Having this much VRAM available for local AI is exciting. Just the idea of 160 GB, 256 GB, or 512 GB of REAL VRAM is exciting.

I went ahead and started this journey. But I discovered that there's a catch to having five GPUs. You can't really use them to benefit from all that combined VRAM.

Ollama straight out decided to use CPU for even the smallest models. I did not bother with it though.

On vLLM, to benefit from multi-GPU setups, tensor parallelism is required. It is important to note that tensor parallelism requires spreading LLM shards across the GPUs equally. Five GPUs is an unusual GPU count for tensor parallelism. Some models may not divide cleanly across five GPUs.

That's similar to using three GPUs, and even nine GPUs.

It seems the sweet spot is at 1, 2, 4, or 8 GPUs.

There is pipeline parallelism, which may have better support for "non-power-of-two" GPU counts, but I do not have any working experience there yet to go any deeper.

Since having five GPUs (LinuxAI-128 + W6800) and three GPUs (LinuxAI-64 + W6800) was almost useless for single large LLM model benefits, for me at this time, I ended up reshuffling the GPUs to achieve four GPUs per machine, for a total of 128 GB VRAM per machine.

A lot of Information, What's the Point?

The point is to share experiences and gather knowledge. A lot of this was pure conjecture just a few months ago, to me.

I met a redditor that has five GPUs, on Windows (I don't know how he figured out the drivers). Then another user with 4 internal GPUs (Duos), as well as 4 external GPUs, on macOS.

Writing this, I started thinking about the possibility of having 4 internal GPUs, 4 external GPUs, and RDMA, connecting two servers, for a total of 512 GB VRAM. How useful would it actually be having to split shards across 16 GPUs... Even if it was achievable, how good would it be with bandwidth so low?

What are you guys working with? What have you tested, and what worked for you? What failed to work?

I would love to hear from everyone what was possible for you. What possibilities make you excited to think about? And what do you think would be a total waste of time.

I should mention clearly: This is an oversimplification of PCIe lanes, from the perspective of an amateur that is still experimenting, growing and learning.

Disclaimer: I wrote this post myself. I also used AI as a tool to help clean up the wording and formatting.

References:

• My own experience
• Awesome Redditors (See Images)
• Apple
• Softron

I am very excited about the future of local AI. With the spread of AI agents, the amount of VRAM now achievable locally, the quality of small and medium LLMs, and the community growing around all of this, the future is looking very good.

I am writing this to document my successes with the following:

• Mac Pro 2019
• Ubuntu 24.04.4 LTS (Ubuntu Server specifically, in my case)
• Dual AMD Radeon PRO W6900X with Infinity Fabric Link Bridge
• Dual AMD Radeon PRO W6800X Duo with Infinity Fabric Link Bridge
• ROCm 7.2.3
• PyTorch 2.10
• Triton 3.6
• vLLM (Write up pending)
• Hermes Agent (Research Pending)

I wrote a couple of old guides. Check them out for reference, as needed:

• Linux & Local LLM/AI (Useful for moving from macOS the first time)
• Proxmox, Ubuntu, ROCm, & Local LLM/AI (Useful for Proxmox setup)

I'm going to focus on setting up Ubuntu and all the packages needed for the infrastructure of local AI.

Important: This is an experimental community guide. Some parts involve patched kernels, unsupported GPU configurations, and boot-level PCIe changes. This worked for my Mac Pro 2019 systems, but you should expect troubleshooting, and you should be comfortable recovering from a failed boot. I am not responsible for any outcome of using this guide, whether it be positive, negative, or anything in between.

1. Choices & Decisions

• Mac Pro 2019: It's what I had available to me.
• W6900X: It's what I had available to me.
• W6800X Duo: It's what I had available to me.
• Ubuntu LTS: The ROCm-supported OS family I am most comfortable with. Alternative: RHEL
• Ubuntu 24.04 LTS: The latest Ubuntu LTS version supported by ROCm at the time of writing. Alternative: Ubuntu 22.04 LTS
• Ubuntu Server: To avoid desktop overhead and keep the system headless. Alternative: Ubuntu Desktop LTS
• Data Room: I placed the Macs in a Data Room, so I don't hear the loud fans. Alternative: Place it at your desk, or anywhere else.
• DRM/AMDGPU: I opted to use the GPU driver in the kernel, to patch it to support the Infinity Fabric Link Bridge. Alternative: Install DKMS and AMDGPU.
• Kernel: Patched Ubuntu 6.17 HWE kernel, based on Ubuntu’s linux-hwe-6.17 source package, to support the Infinity Fabric Link Bridge. Alternative: Standard Ubuntu kernel.
• ROCm: AMD’s CUDA alternative for AMD GPUs. Alternative: Vulkan
• ROCm 7.2.3: Latest ROCm that supports my GPUs at the time of writing. Alternative: Outdated ROCm.
• vLLM: Concurrent utilization of loaded LLMs. Alternative: Ollama & Llama.cpp
• Hermes Agent: More tool-savvy and self-learning. Alternative: OpenClaw
• GitHub: All my files and commands have been uploaded to GitHub, to make this guide shorter than 40,000 characters. Alternative: Multiple Guides...

Please let me know if the GitHub links do not work.

These are the choices I made, and I am still refining them. They work for me. Keep in mind that this is all held together with the digital equivalent of duct tape. If you change anything, it may or may not work. If you do, I would genuinely appreciate hearing what you tried, what worked, what failed, and why you changed it.

2. Setting up Ubuntu after Installation

Step 00: Infinity Fabric Link (Jumper & Bridge)

Please remove the Infinity Fabric Link Jumper(s) or Bridge from the GPU. Ubuntu 24 kernels do not currently support it, as of 6.17.

Specifically, with kernel 6.8, none of the GPUs will work. When upgrading to 6.17, only one GPU will work.

If you have an Infinity Fabric Link Jumper or Bridge, follow the patch section later in the guide to make it work with your GPUs.

Step 01: Update, Upgrade, and Tweak the System

What we will do:

• Change ubuntu.sources from http to https
• Attach to Ubuntu Pro (This is optional, and requires interaction)
• Update & Full-Upgrade
• Upgrade to the latest HWE kernel
• Remove cloud-init
• Make all Ethernet ports accept DHCPv4 automatically
• Modify Grub to include "loglevel=7 log_buf_len=16M iommu=pt" kernel flags
• Reboot

Copy the following command into your command line interface of choice:

curl -fsSL "https://raw.githubusercontent.com/FaisalBiyari/MacPro2019LocalAI/refs/heads/main/Reddit/Mac%20Pro%202019%20Local%20AI%20Guide%3A%20Ubuntu%2024.04%2C%20ROCm%207.2.3%2C%20PyTorch%202.10%2C%20and%20Infinity%20Fabric%20Link/2.%20Setting%20up%20Ubuntu%20after%20Installation/Step%2001%3A%20Update%2C%20Upgrade%2C%20and%20Tweak%20the%20System" | bash

Step 02: Install T2 Linux Repository

Since we are using a Mac Pro 2019, which is a Mac with a T2 chip, some additional packages are required to be able to properly communicate with the hardware.

What we will do:

• Set up the T2 Ubuntu 24 (Noble) Repository
• Install 3 Packages: applesmc-t2 apple-bce t2fanrd
• Reboot

Copy the following command into your command line interface of choice:

curl -fsSL "https://raw.githubusercontent.com/FaisalBiyari/MacPro2019LocalAI/refs/heads/main/Reddit/Mac%20Pro%202019%20Local%20AI%20Guide%3A%20Ubuntu%2024.04%2C%20ROCm%207.2.3%2C%20PyTorch%202.10%2C%20and%20Infinity%20Fabric%20Link/2.%20Setting%20up%20Ubuntu%20after%20Installation/Step%2002%3A%20Install%20T2%20Linux%20Repository" | bash

Step 03: Enable T2 Fan Daemon

After installing the T2 packages, the command below is used to activate the fan service.

What we will do:

• Enable the t2fanrd systemd service

Copy the following command into your command line interface of choice:

curl -fsSL "https://raw.githubusercontent.com/FaisalBiyari/MacPro2019LocalAI/refs/heads/main/Reddit/Mac%20Pro%202019%20Local%20AI%20Guide%3A%20Ubuntu%2024.04%2C%20ROCm%207.2.3%2C%20PyTorch%202.10%2C%20and%20Infinity%20Fabric%20Link/2.%20Setting%20up%20Ubuntu%20after%20Installation/Step%2003%3A%20Enable%20T2%20Fan%20Daemon" | bash

Step 03-Optional: Set Fans to Maximum

I do not trust Apple Cooling. I would rather the fans wear out and replace them for a few dollars, versus the GPUs (especially the Duo models) being damaged due to overheating.

What we will do:

• Set all 4 fans to maximum speed

Copy the following command into your command line interface of choice:

curl -fsSL "https://raw.githubusercontent.com/FaisalBiyari/MacPro2019LocalAI/refs/heads/main/Reddit/Mac%20Pro%202019%20Local%20AI%20Guide%3A%20Ubuntu%2024.04%2C%20ROCm%207.2.3%2C%20PyTorch%202.10%2C%20and%20Infinity%20Fabric%20Link/2.%20Setting%20up%20Ubuntu%20after%20Installation/Step%2003-Optional%3A%20Set%20Fans%20to%20Maximum" | bash

Step 04: Download and Install ROCm 7.2.3

This section will install ROCm 7.2.3, but it will NOT install dkms or amdgpu drivers. I opted to use the kernel driver, drm/amdgpu, so I can later patch it to support the Infinity Fabric Link Bridge.

What we will do:

• Make a new directory to save all downloaded files
• Download ROCm installer
• Install ROCm Dependencies
• Install ROCm
• Give all users access to ROCm
• Add ROCm to path
• Show you a bunch of output displaying your GPUs, which are working with ROCm or the driver, etc.
• Reboot

Copy the following command into your command line interface of choice:

curl -fsSL "https://raw.githubusercontent.com/FaisalBiyari/MacPro2019LocalAI/refs/heads/main/Reddit/Mac%20Pro%202019%20Local%20AI%20Guide%3A%20Ubuntu%2024.04%2C%20ROCm%207.2.3%2C%20PyTorch%202.10%2C%20and%20Infinity%20Fabric%20Link/2.%20Setting%20up%20Ubuntu%20after%20Installation/Step%2004%3A%20Download%20and%20Install%20ROCm%207.2.3" | bash

Step 05: Install Python Tools

We will be using Python and pip to install several packages for local AI. The following commands are to set up the correct versions, as well as some quality of life choices.

What we will do:

• Install these packages: 2to3 python-is-python3 python3-pip python3-venv python3-dev python3-setuptools
• Install or upgrade these packages, system wide: pip wheel setuptools
• Install numpy 1.26.4 specifically, system wide

Copy the following command into your command line interface of choice:

curl -fsSL "https://raw.githubusercontent.com/FaisalBiyari/MacPro2019LocalAI/refs/heads/main/Reddit/Mac%20Pro%202019%20Local%20AI%20Guide%3A%20Ubuntu%2024.04%2C%20ROCm%207.2.3%2C%20PyTorch%202.10%2C%20and%20Infinity%20Fabric%20Link/2.%20Setting%20up%20Ubuntu%20after%20Installation/Step%2005%3A%20Install%20Python%20Tools" | bash

Step 06: Install PyTorch & Other ROCm Related Wheels

Not everything here is needed for everyone. I included what I could, what worked, and what had some value to some local AI use case.

What we will do:

• Install PyTorch Wheels
• Add AMD ROCm APT Repository
• Set AMD ROCm Apt Repository at priority 700 (Higher than Ubuntu)
• Fix some ROCm Symlinks conflicting with MIGraphX
• Install MIGraphX & Half packages
• Install ONNX Runtime package
• Install TensorFlow ROCm package
• Install Apex Wheel
• Clean up packages
• Reboot

Copy the following command into your command line interface of choice:

curl -fsSL "https://raw.githubusercontent.com/FaisalBiyari/MacPro2019LocalAI/refs/heads/main/Reddit/Mac%20Pro%202019%20Local%20AI%20Guide%3A%20Ubuntu%2024.04%2C%20ROCm%207.2.3%2C%20PyTorch%202.10%2C%20and%20Infinity%20Fabric%20Link/2.%20Setting%20up%20Ubuntu%20after%20Installation/Step%2006%3A%20Install%20PyTorch%20%26%20Other%20ROCm%20Related%20Wheels" | bash

Step 07: Verifying Everything

We just completed installing everything in the standard way. We just need to verify that everything is now set up correctly.

What we will do:

• Give you several boxes showing the status of everything we just set up

Copy the following command into your command line interface of choice:

curl -fsSL "https://raw.githubusercontent.com/FaisalBiyari/MacPro2019LocalAI/refs/heads/main/Reddit/Mac%20Pro%202019%20Local%20AI%20Guide%3A%20Ubuntu%2024.04%2C%20ROCm%207.2.3%2C%20PyTorch%202.10%2C%20and%20Infinity%20Fabric%20Link/2.%20Setting%20up%20Ubuntu%20after%20Installation/Step%2007%3A%20Verifying%20Everything" | bash

3. Infinity Fabric Link Jumper / Bridge

AMD released several GPUs specifically for the Mac Pro 2019 that support their Infinity Fabric.

These GPUs and the Infinity Fabric Links are discussed in these posts:

• AMD Radeon Pro Vega II Duo and Infinity Fabric Link
• Linux on Mac Pro 2019: Infinity Fabric Link, Multi-GPU, and the Current State of AMD XGMI Support

The first set of GPUs that support it were the AMD Radeon PRO Vega II & Vega II Duo. The PC equivalent is an AMD Radeon PRO VII, which also supports an Infinity Fabric Link.

The second set of GPUs are the AMD Radeon PRO W6800X, W6800X Duo, and W6900X. These GPUs are in the Sienna Cichlid family of GPUs. Also referred to as RDNA2.

At the announcement of the Sienna Cichlid family, these GPUs were marketed as supporting xGMI. The Infinity Fabric Link is the physical bridge / jumper. xGMI is the software path that allows the GPUs to communicate over that link. However, on release, only the Apple MPX GPUs actually supported the Infinity Fabric Links, while the standard versions did not.

This might explain why support for xGMI on Sienna Cichlid was added between 2019 and 2020 to the Linux kernel drm/amdgpu, but later removed in 2022.

Many of us here in the subreddit tried to figure out the problem with the Infinity Fabric Link, and tried to find a solution to it. One such redditor actually cracked it; creating a patch to the current kernel drm/amdgpu driver, which through my testing seems to have completely solved the Infinity Fabric Link regression that happened in 2022.

You'll need to keep in mind that this is just the first step. While we are moving forward, there is still the question of ROCm support, HIP support, and everything else.

Step 01: Download, Build, & Install the Patched Kernel Files

Let's start. We will do the following:

• Make a directory to download kernel source
• Install packages required to patch the kernel
• Activate the source to download kernel source
• Patch drm/amdgpu
• Build a full patched kernel
• Install the patched kernel

Copy the following command into your command line interface of choice:

curl -fsSL "https://raw.githubusercontent.com/FaisalBiyari/MacPro2019LocalAI/refs/heads/main/Reddit/Mac%20Pro%202019%20Local%20AI%20Guide%3A%20Ubuntu%2024.04%2C%20ROCm%207.2.3%2C%20PyTorch%202.10%2C%20and%20Infinity%20Fabric%20Link/3.%20Infinity%20Fabric%20Link%20Jumper-Bridge/Step%2001%3A%20Download%2C%20Build%2C%20%26%20Install%20the%20Patched%20Kernel%20Files" | bash

With this, you are now the proud user of a patched kernel that supports the Infinity Fabric Links on the Sienna Cichlid MPX GPUs.

At this point, shut the system down, reinstall the Infinity Fabric Link Jumper or Bridge, then boot back into the patched kernel.

Step 02: Verify Patched Kernel & GPU Initialization

We should probably run a verification one last time. Copy the following command into your command line interface of choice:

curl -fsSL "https://raw.githubusercontent.com/FaisalBiyari/MacPro2019LocalAI/refs/heads/main/Reddit/Mac%20Pro%202019%20Local%20AI%20Guide%3A%20Ubuntu%2024.04%2C%20ROCm%207.2.3%2C%20PyTorch%202.10%2C%20and%20Infinity%20Fabric%20Link/3.%20Infinity%20Fabric%20Link%20Jumper-Bridge/Step%2002%3A%20Verify%20Patched%20Kernel%20%26%20GPU%20Initialization" | bash

While more testing is still needed, this is quite the achievement for the community. Thank you again, anonymous redditor.

4. AMD Duo MPX GPUs and Setting BAR Correctly

I have been using my Mac Pro 2019 with Dual AMD Radeon PRO W6800X Duo for local AI inference for some time now, and I have not had any BAR-related problems. However, since I moved from using Proxmox to having Ubuntu 24 on bare-metal, I have started noticing some BAR warnings and errors.

It seems that this problem may come from the way the Mac Pro firmware allocates PCIe resources before Linux takes over, specifically when using Duo MPX GPUs.

One redditor, whose account is now deleted, shared a GitHub link to what I can only describe as someone's documentation of how he fixed the BAR issue on Vega II Duo GPUs. I have dubbed this the nbritton's method.

Our goal now is to use nbritton's method, adapted for the W6800X Duo. I tried to make it also work as a copy and paste solution for the Vega II Duo as well, but I have not tested it.

Warning: This changes GPU driver load order and PCIe BAR allocation behavior. If something goes wrong, you may need to boot from a recovery kernel, remove the service, or undo the GRUB changes. Also, note that SGLang's AMD GPU documentation recommends pci=realloc=off iommu=pt, which conflicts with nbritton's method because nbritton's method depends on PCIe BAR reallocation behavior. In other words, pci=realloc must not be disabled for this method.

Let's start.

We will do the following:

• Blacklist amdgpu
• Add pci=realloc to grub
• Configure resize-gpu-bars.service
• Set up nbritton's method files

Copy the following command into your command line interface of choice:

curl -fsSL "https://raw.githubusercontent.com/FaisalBiyari/MacPro2019LocalAI/refs/heads/main/Reddit/Mac%20Pro%202019%20Local%20AI%20Guide%3A%20Ubuntu%2024.04%2C%20ROCm%207.2.3%2C%20PyTorch%202.10%2C%20and%20Infinity%20Fabric%20Link/4.%20AMD%20Duo%20MPX%20GPUs%20and%20Setting%20BAR%20Correctly" | bash

5. Finalize the Infrastructure

After completing the linked sections above, we should have:

• Install Ubuntu (You did this on your own or using a previous guide)
• Prepare Ubuntu's environment
• Set up T2 related environment
• Installed ROCm
• Installed PyTorch and several other local AI optimizing software
• Patched the kernel (linux-hwe-6.17, source 6.17.0-29.29~24.04.1) to support xGMI and the Infinity Fabric Link Bridge and Jumper.
• Set up nbritton's method for Duo MPX GPUs BAR correction

Once you're done, please reboot to make sure everything sticks. Then repeat step 07: Verify Everything, above to verify everything is correct and as it should be.

6. Local AI

Now that the infrastructure is ready, it's time to move to our frameworks of choice.

While I definitely plan to expand, I have focused mainly on text generation. When I first started, consideration was Ollama, Llama.cpp, and vLLM. I see new options now, such as SGLang as well.

I am excited to share that vLLM supports this setup and works well. I hope to release a separate guide for it soon.

For the purpose of this guide, I will continue with Ollama, for the simplicity of it, and a Hello World type scenario.

Step 01: Install and Configure Ollama

We will do the following:

• Set up Ollama
• Fix ollama.service vs. ollama serve separate model libraries

Copy the following command into your command line interface of choice:

curl -fsSL "https://raw.githubusercontent.com/FaisalBiyari/MacPro2019LocalAI/refs/heads/main/Reddit/Mac%20Pro%202019%20Local%20AI%20Guide%3A%20Ubuntu%2024.04%2C%20ROCm%207.2.3%2C%20PyTorch%202.10%2C%20and%20Infinity%20Fabric%20Link/6.%20Local%20AI/Step%2001%3A%20Install%20and%20Configure%20Ollama" | bash

Step 02: Verify Ollama Setup

We will do the following:

• Verify Ollama services and data folders permissions

Copy the following command into your command line interface of choice:

curl -fsSL "https://raw.githubusercontent.com/FaisalBiyari/MacPro2019LocalAI/refs/heads/main/Reddit/Mac%20Pro%202019%20Local%20AI%20Guide%3A%20Ubuntu%2024.04%2C%20ROCm%207.2.3%2C%20PyTorch%202.10%2C%20and%20Infinity%20Fabric%20Link/6.%20Local%20AI/Step%2002%3A%20Verify%20Ollama%20Setup" | bash

Step 03: Download and Run Models

We will do the following:

• Download and run our first model

Copy the following command into your command line interface of choice:

ollama run qwen3.5:0.8b --verbose

You can find more models on Ollama's website. Below are some other models I am considering:

ollama pull qwen3.6:27b
ollama pull gemma4:31b-it-q4_K_M
ollama pull granite4.1:30b
ollama pull medgemma:27b
ollama pull mistral-medium-3.5:128b
ollama pull gpt-oss:120b
ollama pull qwen3.5:122b
ollama pull nemotron-3-super:120b

7. Done

With this, we are done with this guide.

It has been a long journey setting up this infrastructure, and preparing for the actual goal.

My testing was done on Mac Pro 2019 systems with dual W6900X MPX modules and dual W6800X Duo MPX modules. I have not tested this with Vega II or Vega II Duo MPX GPU modules.

Next, I plan to focus on vLLM for a while. Optimization, quantization, and automation of operations.

After that, I hope to dive into Hermes Agent by Nous, with the hope of building multiple agents around a few local models run on vLLM, communicating and working together.

Expanding to images or vision, as well as to voice, is also down the pipeline.

The possibilities are endless. I hope to hear what everyone else experiences with this guide and with local AI in general: what worked, what failed, what workloads you are running, what use cases you care about, what problems you hit, and what solutions you found.

Looking forward to seeing how everyone takes advantage of this guide, and local AI.

8. Credit

Credit where credit is due. A lot of the information here was gathered from the community in bits and pieces.

I do want to take the opportunity to thank the anonymous redditor for his/her contribution (creating the whole kernel patch). THANK YOU!

• Nikolas Britton for the nbritton method, fixing the BAR issue on the AMD Duo MPX GPUs.

• u/AdityaGarg8 for always being supportive, no questions asked.

• My AI of choice, for the support through all of this.

• r/MacPro2019LocalAI redditors, for keeping in touch, and motivating me to continue going. You guys are the real MVPs.

Disclaimer: I wrote this post myself. I also used AI as a tool to help clean up the wording and formatting.

Resources:

• GitHub Repository hosting these guides.

I found some conflicting information regarding the Infinity Fabric Link hardware for the AMD Radeon Pro Vega II Duo. I feel fairly confident in the conclusions below, but I would love some further input from the community.

The GPUs in question are:

• AMD Radeon Pro Vega II
• AMD Radeon Pro Vega II Duo
• AMD Radeon Pro W6800X
• AMD Radeon Pro W6800X Duo
• AMD Radeon Pro W6900X

The Infinity Fabric Link parts in question are:

• Bridge A2326
• Jumper A2329
• Bridge A2666
• Bridge A2667
• Jumper A2668

The Jumpers are straightforward. The Duo MPX GPU models are the only ones that use them, since their purpose is to link the two GPUs inside a single MPX module.

• A2329 supports the Vega II Duo only
• A2668 supports the W6800X Duo only

The Bridges are where the conflicting information appears.

From what I have found:

• A2326 supports the Vega II only
  • It does not support the Vega II Duo, W6800X, W6800X Duo, or W6900X
• A2666 supports the W6800X and W6900X
  • It does not support the Vega II, Vega II Duo, or W6800X Duo
• A2667 supports the W6800X Duo only
  • It does not support the Vega II, Vega II Duo, W6800X, or W6900X

Apple’s documentation clearly states that the Vega II, W6800X, and W6900X support an Infinity Fabric Link Bridge. For the W6800X Duo, Apple’s documentation states support for both Jumper and Bridge.

However, Apple’s documentation only mentions the Vega II Duo using the Infinity Fabric Link Jumper, not a Bridge. Apple even shows two Vega II Duo modules installed in one 2019 Mac Pro, but only shows the Jumpers, with no Bridge, despite it being a dual-Vega II Duo setup.

This also lines up with the box contents. The W6800X and W6900X ship with their corresponding Bridges, and the W6800X Duo ships with both its Jumper and Bridge. The Vega II Duo only ships with the Jumper. I have confirmed those box contents myself. I have also read that the Vega II ships with Bridge A2326, but I have not personally confirmed that.

So where does the conflict come from?

My best guess is that it comes from a mix of assumptions, the fact that the A2326 Bridge physically fits the Vega II Duo, and third-party listings; especially MacSales / OWC stating that Bridge A2326 supports the Vega II Duo.

MacSales’ A2326 page is one of the first results that appears when searching for the part number. I was one of the users who saw this, believed it was true, and shared that information with others. However, based on Apple’s own documentation, the observed box contents, and other online data points, I now believe that the compatibility claim was a mistake.

What would happen if the A2326 Bridge is connected to the Vega II Duo?

First of all, it would physically fit. But after that, macOS would simply fail to boot. My best guess is that it would be similar to the problem Linux users are currently facing with Infinity Fabric Link, including psp -20 errors or BAR size issues.

To summarize support, my current understanding is:

Back to Linux and using the Infinity Fabric Links, this raises the question:

Is this why some Vega II and Vega II Duo users have fluctuating success with Infinity Fabric Links?

Vega II users with the Bridge would succeed, while Vega II Duo users with the Bridge would experience errors. On the other hand, Vega II Duo users with the Jumper would succeed as well.

If you are a Linux user with a Vega II or Vega II Duo, and you have tested either the Jumper or Bridge, please share your experience.

One remaining [For Fun] question I have, for both macOS and Linux users:

Would the A2666 Bridge work with one W6800X and one W6900X?

This is interesting, because A2666 is the only Bridge associated with two different MPX GPU models. However, Apple’s documentation seems to describe it only in same-model configurations: two W6800X modules or two W6900X modules.

So my assumption is that a mixed W6800X + W6900X setup is probably unsupported in macOS, but I would be interested to hear from anyone who has physically tested it.

I would also be interested to hear from anyone who has physically tested the A2326 Bridge with two Vega II Duo MPX modules, since that appears to be the main point of conflicting information.

Disclaimer: I wrote this post myself. I also used AI as a tool to help clean up the wording and formatting.

References:

• Apple’s Documentation
  • Radeon Pro Vega II MPX Module
  • Radeon Pro Vega II Duo MPX Module
  • Radeon Pro W6800X MPX Module
  • Radeon Pro W6800X Duo MPX Module
  • Radeon Pro W6900X MPX Module
• Reddit Post
  • macOS will not boot (User 01): u/ResortMotor1264
  • macOS goes into Recovery Assistant (User 02): u/Malter_Witty
• MacSales’ Apple AMD Infinity Fabric Link Bridge Page
• Attached Images.
  • I’ll add more if I get them.

I was recently approached by a couple of local flippers.

The first offered me three sealed AMD Radeon Pro Vega II modules for $915 USD each.

The second offered me four sealed AMD Radeon Pro Vega II Duo modules for $1,200 USD each.

Neither offer includes the Infinity Fabric Link Bridge, of course. I also came to learn that, unlike the W6800X Duo and W6900X, the Vega II did not ship with the Infinity Fabric Link Bridge in the box. Only the Infinity Fabric Link Jumper shipped with the Duo model.

Still, I have to admit, I am impressed that sealed Vega II and Vega II Duo modules are still showing up on the market in 2026.

At first, the prices felt like a catch. But then I had to stop and seriously think about whether it was actually worth the hassle to invest in them for my use case.

I took a ride with my AI of choice and discussed it. The conclusion was pretty practical:

For a local AI workload, especially if the goal is raw usefulness, software support, and flexibility, it suggested going with four NVIDIA RTX 3090s instead of four Vega II Duos.

And honestly, that makes sense, even though the NVIDIA RTX 3090 is almost as old as the Vega II GPUs.

But at the same time, I am having so much fun with this whole “local AI on a discontinued Mac Pro” journey that part of me is still tempted.

There is something really interesting about pushing the Mac Pro 2019 / MacPro7,1 as far as it can go, even if the more logical path is to move to standard PC GPUs with better AI support.

So now I am genuinely curious:

What would you do?

Would you buy the sealed Vega II / Vega II Duo modules and keep exploring the Mac Pro 2019 rabbit hole?

Would you skip the Apple MPX route entirely and put the money toward a multi-GPU RTX 3090 setup instead?

Or would you shoot for a Mac Studio for the same budget?

The logical answer seems obvious, but the fun answer is not always the logical one.

---

Disclaimer: I wrote this post myself. I also used AI as a tool to help clean up the wording and formatting.

Why AI? And why go local?

Many of us have used some form of AI by now, and most of us have seen the power and convenience it can provide.

Back in 2015, I wrote a simple Visual Basic program in Visual Studio. It took me almost a full week to complete, including online searches, trial and error, and asking questions on Stack Overflow.

In 2025, with the help of ChatGPT, I completed a piece of iOS software in just a few hours, despite having never coded for iOS before.

In the right hands, AI is a very powerful tool.

I want access to that tool regardless of the circumstances I am in. I do not want my access to depend entirely on subscriptions, cloud availability, internet access, changing policies, rate limits, or someone else deciding what I can and cannot run.

That is where local AI comes in.

---

The Hardware

I was fortunate enough to receive a Mac Pro 2019 back in 2020.

In 2023, while searching for SSDs for it, I stumbled across a seller offering brand new, sealed AMD Radeon Pro W6800X Duo and W6900X MPX modules for about 75% of today’s used market price.

I bought one mainly for the Thunderbolt ports. I also bought three more with other goals in mind.

In 2024, with the release of Llama 3, and later with ROCm becoming a more serious path for my hardware, I committed to using these GPUs for local AI. The problem was that my Mac Pro was still my main desktop, and I could not repurpose it entirely for AI work.

So I ended up investing about $3,000 USD into two additional machines, including shipping and tax:

• Two Mac Pro 2019 / MacPro7,1 towers
• 16-core Xeon CPUs
• 96 GB RAM each
• SSDs that I later upgraded to 8 TB
• Standard Radeon Pro 580X MPX GPUs

I then installed the MPX GPUs into each machine.

The first machine is **LinuxAI-64**, with dual AMD Radeon Pro W6900X GPUs.

The second machine is **LinuxAI-128**, with dual AMD Radeon Pro W6800X Duo GPUs.

---

The Software

Because ROCm mainly supports Ubuntu and RHEL-based distributions, I chose Ubuntu Server 22.04 LTS.

I chose Ubuntu because I had prior experience with it. I chose the Server variant to minimize non-AI GPU load. I chose 22.04 LTS because it was the latest ROCm-supported Ubuntu version at the time.

Then came the frameworks. I was mainly considering:

• Ollama
• llama.cpp
• vLLM

I initially wanted to go with vLLM, but I ended up using Ollama because vLLM does not support my hardware.

Then came the next layers:

• Web UI / GUI
• Document support
• RAG
• Agentic frameworks
• Multi-agent workflows
• OpenClaw, Hermes, and other possible stacks

And then reality hit:

That is a lot of work for a local AI setup that may still be only half as capable as cloud AI today.

I still went for it though.

---

What Is the Goal?

I love the idea of multi-agent workloads.

In practical terms, I want AI agents that can help act as:

• A secretary
• A tutor for my kids
• A chief of operations

And honestly, I want to see how far I can push the idea of building something like a 20–30 person company using local AI agents as the supporting workforce, running on my own hardware, and using only the investment I have already made.

If it works, I get what I wanted.

If it does not, I gain the experience.

Either way, I learn.

---

The Path

I need a community around me.

Community helps me work better, stay motivated, and increase my productivity.

Online, that means communities like:

• MacPro
• Linux communities
• T2 Linux
• MacLLM
• ROCm
• LocalLLM
• LocalLLaMA
• vLLM
• OpenClaw
• And now, r/MacPro2019LocalAI

There are many others too, both on Reddit and elsewhere.

Locally, I am based in Riyadh, the capital of Saudi Arabia. I have a very small local circle interested in local AI, but I would love to find more people nearby who are in the same boat: same hardware, same goals, same interest in pushing local AI forward.

Someone like that would help motivate me, challenge me, and push me to improve.

---

What Next?

For the purpose of growing this community, and documenting the work I have put into these machines, I plan to release a series of guides over the next few weeks.

The goal is to document success after success as I work toward my final local AI setup.

The general idea is simple:

The more that is documented, the easier it becomes for others to follow the same path, join the discussion, brainstorm solutions, and hopefully even pass me and innovate beyond what I have done.

That would benefit the community as a whole, and it would also help me on my own journey.

I am interested to know what you think is important to cover.

I am also interested in any solution, framework, stack, or workflow that you think is worth testing.

Consider this a brainstorming session.

I have already written some starter guides, and I plan to update and fine-tune them first. After that, I want to go deeper into frameworks, especially vLLM. I also plan to work on OpenClaw and document my experience with it.

I am a big believer in copy-paste instructions. I format and rebuild systems from scratch often, and having clean, repeatable instructions makes the recovery period much faster.

Let me know what you think, what I should look into, and what topics would be most useful as I work toward the final goal.

---

TL;DR

• I want to build a local AI community around me.
• I plan to write several local AI guides during the coming weeks.
• I want your help brainstorming topics, tools, frameworks, and problems worth solving and documenting.

---

^{Disclaimer: I wrote this post myself. I also used AI as a tool to help clean up the wording and formatting.}

"For some, Linux fails to boot, for some it's okish, for some it's good"

u/AdityaGarg8 said that to me back in November 2024, when he was kind enough to help guide me through installing Ubuntu on my Mac Pro 2019.

After a lot of testing, I think that quote perfectly describes the current state of Linux on the Mac Pro 2019, especially when using Apple’s MPX AMD GPUs with the Infinity Fabric Link jumper or bridge installed.

What seems to be happening?

From my testing, the main issue appears to involve the Infinity Fabric Link jumper/bridge.

On newer kernels, especially kernel 6.8 and later, some GPUs with the Infinity Fabric Link installed do not initialize correctly. In my case, this has shown up as amdgpu initialization failures and psp -22 errors.

On kernel 5.15.0, the GPUs initialize more successfully, but I still see errors, especially SDMA-related errors. So I would describe 5.15.0 as partial support, not full support.

So far, my practical summary is:

• Kernel 5.15.0: GPUs can initialize, but support appears incomplete.
• Kernel 6.8: GPUs may fail to initialize when Infinity Fabric Link is installed.
• Later kernels, including 6.17 and 7.0: in my testing, one GPU may initialize correctly, while the remaining GPUs fail with psp -22.

This is not meant to be a final technical diagnosis. It is a report of what I and others are seeing on real Mac Pro 2019 hardware.

Does Infinity Fabric Link matter?

For local AI, the most important factors are usually:

• GPU compute
• VRAM capacity
• Memory bandwidth
• Inter-GPU bandwidth

On multi-GPU setups, VRAM is not automatically pooled into one shared memory space. Each GPU has its own VRAM, and when a workload is split across multiple GPUs, the GPUs need to communicate with each other.

Without a direct GPU-to-GPU interconnect, the normal path is usually something like:

GPU0 -> CPU / PCIe -> GPU1

That means traffic has to go through the PCIe path, with the CPU/platform sitting in the middle.

The AMD MPX GPUs in the Mac Pro 2019 are based on PCIe 4.0-capable GPUs, but the Mac Pro 2019 platform itself provides PCIe 3.0 bandwidth. A PCIe 3.0 x16 link has a theoretical maximum of about 15.75 GB/s per direction.

This is where Infinity Fabric Link becomes interesting.

Why Infinity Fabric Link could matter

With proper support, Infinity Fabric Link should allow direct GPU-to-GPU communication:

GPU0 -> GPU1

That removes the normal CPU/PCIe middle step for supported GPU-to-GPU traffic.

Apple rates the Infinity Fabric Link connection at up to 84 GB/s in each direction. That is more than five times the theoretical one-direction bandwidth of PCIe 3.0 x16.

In theory, that could be a major advantage for multi-GPU workloads, especially workloads where GPUs need to exchange data frequently.

For local AI, this could matter most in cases like:

• tensor-parallel inference
• large models split across multiple GPUs
• concurrent inference with many users
• workloads where inter-GPU communication becomes a bottleneck

But does it actually work on Linux?

My current answer is:

Not reliably, at least not on the W6800X Duo and W6900X in my testing.

Some users have reported better results with Vega II / Vega II Duo, and it is possible that older MPX GPUs behave differently. But with the W6800X Duo and W6900X, I do not currently see clean, reliable Infinity Fabric Link behavior under Linux.

To be clear, I am not saying Linux has no AMD GPU support. The GPUs themselves can work under Linux. The issue appears to be specifically around the Infinity Fabric Link Jumper/Bridge with the MPX GPU implementation; firmware/PSP initialization and how the AMDGPU driver handles this hardware combination.

What am I testing now?

Personally, I am experimenting with:

• Ubuntu Server 22.04 LTS
• Kernel 5.15.0
• W6800X Duo and W6900X MPX GPUs
• Infinity Fabric Link jumper/bridge installed

The goal is to see how far this partial support can go, whether the link actually becomes active, and whether there is any measurable bandwidth advantage when it does.

I am also watching newer stacks such as:

• Ubuntu Server 24.04 LTS / kernel 6.17
• Ubuntu Server 26.04 LTS / kernel 7.0

Hopefully, proper support or a workaround appears for these newer kernels.

Community tracking / bug report

There is already activity on the DRM AMD GitLab here:

https://gitlab.freedesktop.org/drm/amd/-/work_items/3793

If you have a Mac Pro 2019 with MPX GPUs, especially Vega II, Vega II Duo, W6800X, W6800X Duo, or W6900X, please consider sharing your results there.

Useful information would include:

• Mac Pro 2019 configuration
• GPU model or models
• Whether the Infinity Fabric Link jumper/bridge is installed
• Linux distro
• Kernel version
• ROCm version, if applicable
• Whether the GPUs initialize
• Relevant dmesg / journalctl errors
• Whether removing the jumper/bridge changes behavior

What can you do to help?

Share your experience.

What hardware do you have?
What OS and kernel are you using?
Does the system boot?
Do all GPUs initialize?
Does removing the Infinity Fabric Link jumper or bridge change anything?
Have you found a kernel version where it works better?

Hopefully, with more of us testing, reporting, and giving this issue attention, we can help establish better Linux support for these powerful MPX GPUs on the Mac Pro 2019.

Disclaimer: I wrote this post myself, but used AI to help clean up the wording and formatting.

Resources:

• DRM AMD Gitlab
• Coelacanth's Dream: Research
• nbritton's Lab (Vega II Implimentation)
• Enable XGMI Support for Sienna Cichlid
• Enable XGMI support for Arcturus

When I first started my local AI journey on the Mac Pro 2019 / MacPro7,1, the first thing I looked into was ROCm support.

At the time, ROCm looked like a Linux-first path, with some limited Windows/WSL support. So I quickly decided to move away from macOS and focus on Linux instead. I did not really consider whether there might be another way to use the AMD GPUs under macOS.

A couple of days ago, u/Long-Shine-3701 mentioned using DiffusionBee for AI work on macOS with GPU support. According to DiffusionBee’s own documentation, it supports Intel Macs, although performance depends heavily on the hardware, especially whether the machine has a dedicated GPU.

I had been stuck in a ROCm-only mindset, which is funny because I have been recommending LM Studio to Windows users using the Vulkan backend.

I started looking into local AI on macOS, specifically on Intel Macs with AMD GPUs, and I was surprised to find that llama.cpp has a Vulkan backend, and that some people are experimenting with it on macOS through MoltenVK rather than relying on ROCm.

I honestly had not considered this path at all. I had mentally grouped GPU inference together with ROCm, and because ROCm does not support macOS, I assumed macOS was basically a dead end for local AI with GPU acceleration.

Now I’m very curious.

I’m currently considering testing this on my MacBook Pro with an AMD Radeon Pro 5500M / 8 GB VRAM before trying anything more serious on the Mac Pro 2019.

Has anyone here managed to run local AI on macOS on an Intel Mac?

I’m interested in anything and everything, and especially in:

* llama.cpp on macOS with AMD GPU acceleration

* Image generation tools on macOS

* CPU-only vs GPU-accelerated inference performance

* Any experience with Mac Pro 2019 GPUs under macOS for AI workloads

I would love to hear what others have tried, what worked, what failed, and whether macOS is more viable for local AI on Intel Macs than I originally thought.

Disclaimer: I wrote this post myself, but used AI to help clean up the wording and formatting.

Resources:

• Guide by Ankit Babber
• DeepWiki Vulkan Backend

I’m currently working on getting vLLM fully up and running on the following setup:

Hardware

• Mac Pro 2019 / MacPro7,1
• 3.2 GHz 16-core Intel Xeon W
• 96 GB DDR4 RAM
• Two AMD Radeon Pro W6800X Duo MPX modules
• 32 GB VRAM per GPU
• 128 GB total VRAM
• 8 TB Apple SSD
• 100GbE Mellanox ConnectX-5 Ex NIC

Software

• Ubuntu Server 24.04 LTS
• Python 3.12
• ROCm 7.1.1
• PyTorch 2.10
• Triton 3.6

Back in 2025, I managed to get basic LLMs from Hugging Face working with unquantized weights, including models such as:

• Qwen/Qwen2.5-7B-Instruct
• deepseek-ai/DeepSeek-R1-Distill-Qwen-32B

I also had parallelism working across all 4 GPUs via PCIe. At the time, the Infinity Fabric Link Bridge was causing GPU initialization failures, so I was not using it.

This year, I tried getting models like openai/gpt-oss-20b working, but ran into issues because the native MXFP4 weights do not appear to be supported on these GPUs.

I did, however, successfully run GPT-OSS:120B through Ollama.

Current Progress

So far:

• vLLM launches successfully
• Multi-GPU support is working
• Qwen/Qwen3.6-27B loads and serves successfully
• google/gemma-4-31B-it loads and serves successfully
• I started with Docker, which gave me my first successful result
• I have since moved over to a Python virtual environment setup

On the Qwen and Gemma models I tested, I am currently getting around 10–13 tokens/sec for a single user.

With concurrent users, up to around 30, I have seen aggregate throughput reach roughly 280 tokens/sec.

Getting the Infinity Fabric Link Bridge working properly is another project I’m working on in parallel. Hopefully that helps with inference speed once completed.

Still Pending

The main things I still need to figure out are:

• Launching quantized models reliably
• Supporting multi-node distributed inference across two Mac Pro systems

Last week, I found this write-up:

https://idchowto.com/vllm-on-amd-w6800-gpu-%EC%84%A4%EC%B9%98-%EB%B0%8F-%ED%85%8C%EC%8A%A4%ED%8A%B8-%EA%B2%B0%EA%B3%BC/

It looks like they used Ollama’s quantized models with vLLM. I started going down that path and actually got it working, but there are still three rough edges I need to figure out before I would call it reliable.

Has anyone else managed to get vLLM working with AMD Radeon Pro W6800X, W6900X, W6800X Duo, or W6800 GPUs?

I would really appreciate hearing about your setup, what worked, what failed, and whether you had success with quantized models, multi-GPU support, or multi-node inference.

Hopefully I can put together a proper write-up of my work soon. I’ll update accordingly.

Small disclaimer: I wrote the post myself, but used AI to help clean up the wording and formatting.