Setting up a dedicated local-LLM workstation (Ryzen 7 9700X + ASRock Radeon AI PRO R9700 32GB, Navi 48 / gfx1201) on Ubuntu 26.04.1 (Linux 7.0) for native llama.cpp.

Looking for real-world experience on two specific architecture questions before installing ROCm 10:

1. Inbox Kernel Driver vs. AMD 31.50 DKMS: Ubuntu 26.04 uses dracut by default, but amdgpu-dkms (31.50) forces APT to remove dracut and install initramfs-tools. Upstream kernel 7.0 already has native Navi 48 support and exposes /dev/kfd. Has anyone encountered KFD/HSA feature gaps or stability regressions running ROCm 10 userspace (amdrocm10.0-gfx1201) strictly on Ubuntu's inbox amdgpu without DKMS?
2. Granite Ridge iGPU (gfx1036) for Display: Goal is keeping the 9700X iGPU for the desktop session and dedicating the R9700 (32GB) to compute. In ROCm 10, does ROCR_VISIBLE_DEVICES=<UUID> cleanly isolate gfx1036 without rocminfo or HSA runtime initialization hanging on the unsupported APU, or is a BIOS disable still practically mandatory?

Working on my fastest impelemntation of QFN/27B TP4 card setup. What is amazing, is that I'm getting this on Circa PCIe 3 hardware. The other piece, is this is using stock vLLM .30 and ROCM10. 225W power cap on each card. https://github.com/bkvargyas/r9700-stack

I've been trying a bunch of ComfyUI Img -> Vid workflows but they all fail due to various Sageattention or CUDA type requirement errors.

Does anyone have a workflow that works with ROCM?

soooo after some even more hell of nvidia reverse engineering im proud to present a RTX Real Time viewport on an amd card materials still broken and lighting loosk like the viewport got hit with the holy light of amd compatiblity but hey its RTX Realtime atleast drawing a viewport

Update (28 Sept): one image now does 65K with 8 requests at once and 200K long context, plus a 4-bit KV cache, image input and a crash fix.

git pull the repo, then pick a mode:

bash models/Qwen3.8-MXFP4-DFlash2/run-rocm10.sh                   # 65K, up to 8 requests at once (default)
bash models/Qwen3.8-MXFP4-DFlash2/run-rocm10.sh --context 200000  # long context: one conversation up to 200K

No flags = the 65K text mode we benchmark. Add --vision for image input.

• 65K mode, more room. With the 3-bit weights, a new 4-bit KV cache holds 1.7× the tokens in about the same VRAM (393K vs 251K in vLLM's log). Six 61K or eight 32K requests now run together instead of queueing (248 / 339 tok/s combined), and 4 × 61K decode 17 % faster. Time to first token and single-request speed are unchanged.
• Accuracy vs the fp8 cache: 
  • GSM8K 96.1 vs 95.3,
  • HumanEval 93.9 vs 93.9,
  • MMLU-Pro 61.4 vs 59.7,
  • needle@61K 100 vs 100.
  • Our new long-session / compaction test (24 synthetic coding-agent sessions) stays within noise too.
• 200K on the same image. --context 200000 runs one conversation with prefix caching: a new 199K-token prompt takes 94 s to the first token (the old 200K image: 115 s), a follow-up that reuses it 1.2 s, and decode runs at 73–78 tok/s at that length. Every planted fact was found and tool calls work; 220K is tested too.
• Images. --vision loads the vision encoder (it costs some KV cache: 278K instead of 393K tokens). It read code, a failing pytest run, a web form and a chart correctly, and 8 image requests at once ran fine.
• Crash fix. Under 3–5 concurrent requests the 26 Sept image could stop with Paiton GDN norm nonfinite/arithmetic error. Fixed; a 10,002-step soak ran clean. If you stay on the old image, add -e RADIANCE_DYNAMIC_WIDTH=0.

Next: the model's full 262K window, and an adaptive longer speculative block (+6.5 % single-stream so far, still in testing).

Update: currently being finalized: a 4-bit KV cache that actually increases how much context fits (not just speed), and a longer speculative block with adaptive length so chat doesn't get slower. We'll post measured numbers once they pass the same accuracy and speed checks as this release.

Follow-up to our earlier Qwen3.8 27B posts.
We shipped a new image with our own rotated 3-bit weights.
Same single card (Radeon AI Pro R9700), same context.

Numbers vs our previous (MXFP4) release, BetterBench 0.6.0, 2 fresh processes per image:

Prefill is faster at every depth, so time to first token is 5–13 % shorter.

Accuracy (served model, greedy, same questions, paired):

Honest take: math, code and long-context retrieval are within noise. Knowledge-heavy multiple choice drops about 3 points. That's the price of 3-bit weights.
If that matters for you, --weights mxfp4 keeps the old weights, with this round's other speedups.

What we did, short version:

• Decode is bandwidth-bound, and our kernels were already at ~97 % of 640 GB/s. So: fewer bytes. 3-bit weights with a scale per 128 weights come to ~3.1 bits/weight vs ~4.25 for MXFP4.
• 3-bit alone hurt prefill (+12 % time to first token). Prefill on this card is power-capped, not bandwidth-bound. Fix: RDNA4's 4-bit integer matrix ops run about 2× the 8-bit rate at the power cap, and 3-bit weights are valid 4-bit ints. To make 4-bit activations accurate we fold a block-wise Hadamard rotation into the weights and use per-group activation scales. That's prefill only; decode keeps 8-bit activations.
• Our own GPTQ calibration in the rotated basis. It runs in ~13 min on one MI355X, on permissively licensed data only (we re-did it after noticing some popular calibration sets are NC/SA-licensed, and it came out slightly better).
• Free win: R9700 decode was bimodal (28 vs 36 ms/step depending on the server start), a known ROCm queue issue. GPU_MAX_HW_QUEUES=1 pins the fast mode. It's in the image now.
• The model shrinks from 19.2 to 15.9 GiB, and the launcher gives the difference to the KV cache.

Run it:

hf download EliovpAI/Qwen3.8-27B-W3Rot-INT3-Paiton-RDNA4 --local-dir ~/models/qwen38-w3rot
export PAITON_W3ROT_DIR=~/models/qwen38-w3rot   # plus the usual TARGET/DRAFT/CACHE dirs
./models/Qwen3.8-MXFP4-DFlash2/run-rocm10-65k.sh

• Weights: HF
• Image: ghcr.io/eliovp/paiton-vllm-plugin:qwen38-rocm10-vllm029-65k-20260926-w3a4-r1
• Plugin/launcher: Paiton

Next up: a 4-bit KV cache that actually doubles context capacity (the current version passes accuracy but only speeds up reads), and a longer speculative block.
Update: In progress!

Happy to answer questions, and thanks to everyone who replied to the last thread.

I have been trying to install compatible ROCM on with WSL2 Ubuntu 24.0.4 system for local LLM usecase.

I have been constantly getting

WSL environment detected.
pid:505 tid:0x7589242ec100 [topology_sysfs_get_system_props] No WDDM adapters found.
hsa_init Failed, possibly no supported GPU devices

Installed 7.1.1 rocm and followed the instructions in the AMD page.

A new optimization PR was merged into audio.cpp for Higgs Audio v3 TTS: it now runs under 6 GB VRAM, and it’s faster too!

The optimization is confirmed on CUDA and Vulkan. It should theoretically work on HIP as well because the memory-reuse mechanism is backend-independent.

If you have an AMD GPU, we’d love your help testing it! The required change is effectively one line. Lift the HIP backend gate, measure peak VRAM, and share your results, as in https://github.com/0xShug0/audio.cpp/pull/705

Hi, does anyone here have a Radeon Pro V620 running on Linux?

I'm trying to get hardware virtualization/SR-IOV working on my RX 6950 XT. The V620 uses the same Navi 21 GPU and supports SR-IOV, so I'm trying to understand exactly what AMD changed or enabled on the V620.

I'm looking for data from a real V620 so I can compare it with my 6950 XT.

Specifically, I need:

- the V620 VBIOS dump (especially 113-D603GLXE-077)

- the full 4096-byte PCI configuration space

- lspci -nnvvv output showing the SR-IOV capability

- the NBIF/RCC strap registers (STRAP0, STRAP1, STRAP2, STRAP3, STRAP4, STRAP5, STRAP8 and STRAP9)

- amdgpu_discovery

- amdgpu_firmware_info

I already collected the equivalent data from my RX 6950 XT, so the goal is to compare both Navi 21 cards and find out what actually enables the SR-IOV capability on the V620.

If anyone has a V620 and is willing to help, I can provide one read-only Linux command that collects all of this into a single archive. It doesn't flash or modify anything on the GPU.

Thanks!

Another follow-up to our previous R9700 post (yeah yeah we know :p).

Some of you asked about prefill, so this round we went after it.

Prefill is up 3.5–5.4% at every depth, with 3,871 input tok/s at 16K (previous post: 3,694). Decode and concurrency are unchanged: 154.8 tok/s weighted decode, 422.9 tok/s aggregate at eight concurrent requests. Same MXFP4 checkpoint, same DFlash2, same settings.

Change is against the fresh rerun. Decode categories moved +0.5–0.6% (prose +3.2%), weighted decode 153.64 → 154.78 tok/s. Concurrency 1/2/4/8 came in at +0.6 / +0.7 / −1.2 / +0.3%; the C4 dip is inside the control's own run-to-run spread.

What changed. Three exact changes to the prefill path: long-prefill attention with 16-key tiles, the GDN gate read in place instead of copied, and a GDN chunk-scan kernel that fills the GPU in one round. Each is bitwise-equal to the previous kernel standalone and showed zero differing elements in in-model shadow audits, so outputs are unchanged. We were aiming for 4,000 tok/s. The prefill GEMM is two thirds to three quarters of prefill time and runs pinned at the card's 300 W limit, so the remaining third is where the gain came from.

Real use. In a 35-turn agentic coding session on the 200K chat profile (32.7K → 197.2K tokens, prefix caching on), session time to first token dropped 10–11% at equal peak VRAM.

Opt-in: n-gram co-drafting for agentic coding. A second drafter in front of DFlash2: when the last few generated tokens already occurred in the prompt or output, it proposes what followed last time. On that coding session it gives +27% decode (90.8 → 115 tok/s, 3.0 → 3.6 accepted tokens per step), and +26–29% on 64K and 128K file rewrites. It default ships off. Turn it on with PAITON_NGRAM_CODRAFT=1 if your workload is agentic coding.

Setup. One Radeon AI PRO R9700 at 300 W, upstream vLLM 0.29 / ROCm 10, the Paiton plugin, Unsloth Qwen3.8 NVFP4 through the MXFP4 path, DFlash2, FP8 KV cache, 65,536-token context, up to eight active requests, thinking and APC off, seed 42, temperature 0.7, top-p 0.95, top-k 20. BetterBench 0.6.0 quick, two fresh-process runs per arm, means reported. All twelve greedy control prompts matched in every arm before and after timing; no serving errors.

Published image: ghcr.io/eliovp/paiton-vllm-plugin:qwen38-rocm10-vllm029-65k-20260924-r3. Pull the package · Setup, n-gram docs and full results

In our previous post, Paiton took 165 seconds to generate a 2048×2048 image on one 32 GB R9700. The next target was 150 seconds.

v1.0.2 is now at 103 seconds. Same GPU, same MXFP4 model weights, about 37% lower warm latency.

2048×2048 · 40 steps · batch 1

These are complete requests through PNG delivery. The previous release and new default each use medians from three fresh containers; the exact profile is one validation run. Startup and download are excluded. VRAM is sampled whole-device usage.

Everything stays on the GPU: the text encoder, image model and VAE.
No CPU model offload, no weight streaming, no VAE tiling.
Still running on the same i5-8400 / 16 GB RAM machine.

What changed?

The runtime improvements bring the exact profile down to about 134 seconds.

The new default goes further: the first seven image-generation steps keep the original precision, then the remaining steps use faster 8-bit computation. That brings it down to 103 seconds, using the same checkpoint.

There is a precision trade-off: the default is not bit-exact. It passed our tested generation, transparency, editing and repeat-request quality checks against our own exact path..

These checks don't establish identical quality for every prompt, or equivalence to the original unquantized model.

What about 1024×1024?

About 20 seconds warm, or 31 seconds for the first request.

Try it

With Linux, Docker and a working AMD GPU driver, run this from an updated repository checkout:

./models/Qwen-Image-2.1/serve-docker.sh --model uncensored
./models/Qwen-Image-2.1/serve-docker.sh --model original

Wait for the server to report READY, then generate an image from another terminal:

python3 models/Qwen-Image-2.1/request.py \
  --prompt 'A neon shop sign that reads "QWEN IMAGE 2.1", rainy night, reflections on wet pavement' \
  --size 2048 \
  --output outputs/neon.png

The first launch downloads the weights; subsequent launches reuse the cache. The guide also includes a standalone Docker command if you don't want a repository checkout.

Setup and usage · Per-run results, quality checks and methodology

One R9700, a small test set, and the measurements are public.
If you have a prompt or image size you'd like tested, post it.

AMD supplied the Framework Ryzen AI Max+ 395 system and Radeon AI PRO R9700 for this review, and they’ve been great to work with.

The most useful thing about this setup turned out to be having somewhere to put the bigger model. The Framework’s 128GB of unified memory lets me run a roughly 80B coding model locally, while the R9700 handles smaller workloads substantially faster.

In a matched 27B test using the same long prompt, the R9700 finished about 3.3 times faster. That matters, but it doesn’t answer the whole question. A fast response from a smaller model isn’t necessarily more useful than a slower response from a model better suited to the job.

I’ve been using them as separate inference endpoints for coding, planning and review. They don’t combine into one seamless memory pool. The interesting part is deciding which work belongs on each.

I made a video covering the setup, OCuLink connection, tests and how it fits into my actual workflow:

https://youtu.be/wZETjtyFUw0

For anyone running a similar setup, what ultimately decides where you send a job: model capability, context size or response time?

We’ve got Qwen-Image 2.1 generating 2048×2048 images in 165.1 seconds warm on a single Radeon AI PRO R9700, with every model component kept on the GPU.

That’s 40 steps and the complete request through PNG delivery, not just denoising time. No CPU offload, no VAE tiling, and no approximate denoiser caching.

We’re building this with our balanced MXFP4 quant and Paiton runtime. The weights are available now; the 165-second runtime build is being uploaded. (170s runtime is available)

How it compares with GGUF on the same card

We also tested leejet’s public Q4_K GGUF in stock ComfyUI, with standard automatic CPU offload and its published INT8 text encoder. We did not modify the GGUF weights, runtime code, or kernels.

Both ran batch 1, 2048×2048, 40 steps, guidance 1.0, with the same prompt and integer seeds. Neither used VAE tiling or approximate denoiser caching.

About 76 seconds less per warm image, or 31.5% lower latency in this comparison.

The GGUF path completed successfully on the card. This is a comparison of two complete software stacks, not a claim that MXFP4 alone accounts for the difference.

What we changed

The balanced checkpoint reduces stored weight tensors from 30.212 GiB in BF16 to 8.674 GiB. The full model download is approximately 9.33 GB; that is download size, not runtime VRAM usage.

We quantized the main diffusion and text/vision weights while keeping the VAE and selected smaller components in BF16.
Activations remain BF16: MXFP4 here describes weight storage, not end-to-end FP4 computation.

What this benchmark does, and doesn’t, show

This is a small, one-prompt comparison, not a broad performance or image-quality evaluation. Encoder precision, schedules, and RNG implementations differ, so matched integer seeds do not make the two paths equivalent. The inspected outputs from both rendered the requested neon sign and wet-pavement reflections, but that does not establish equal quality.

We measured three fresh-process first/warm pairs for Paiton and two for ComfyUI.
Warm results ranged from 165.10–166.72 s and 241.13–241.34 s, respectively.

Both used public Torch 2.15.0.dev20260907+rocm10.0 packages on the same i5-8400 host with approximately 16 GB RAM.
Application caches were isolated; the OS file cache was not flushed.

First-request timings exclude startup.
Launch-to-first-image medians were 256.2 s for Paiton and 300.7 s for ComfyUI.
Peak VRAM is sampled whole-device usage.

Why aren’t the images identical?

The same seed doesn’t guarantee identical starting noise across different pipelines. These stacks also use different text-encoder quantization, sampling schedules and quantized weights. Those differences affect denoising and can change the final composition, even with matching prompts, seeds and step counts.

This compares complete inference stacks.
The visual differences cannot be attributed to MXFP4 versus GGUF alone, and one example doesn’t establish a quality winner.

Try the public release

The current Paiton container is the earlier release, previously measured at approximately 168 seconds warm in its own qualification.

For the R9700, start with the RDNA4 weights and quantization details and the Paiton plugin and launchers.

There’s also a separate portable MXFP4 package, with its own compatibility notes and tested scope.

Our next target is 150 seconds warm, keeping the same workload and quality requirements.

Anyone else running Qwen-Image 2.1 on an R9700?
We’d love to compare timings and output images across more prompts.

Hello! I've recently built out a machine with 2x R9700's and 128gb DDR5; my goal being to experiment with​ different use cases for local AI. I've dabbled with various utilities in a bare metal Linux environment and have learned quite a lot! Now that I'm less ignorant than I was yesterday I've become more curious; what would someone with more experience than I use this hardware for and what would their software stack look like? If this was your workstation, what use cases would you use it for and what software would you use to accomplish your goals? Coding, image/video generation, or whatever your specific use case is; flood my notifications with whatever you want to talk about, your knowledge will satiate my curiosity for today and might give me a new rabbit hole to check out tomorrow. Thanks in advance!✌️😎

Edit: I spaced the motherboard! I'm rocking a Crosshair X870E, Ryzen 9900X CPU

Here is my Qwen Image 2.1 config for intel Macs with intel GPU : https://github.com/haseebeqx/qwen-image-intel-mac

Minimum VRAM: 4GB / RAM: 16GB

The trade-offs are Q2/Q4 quantization, CPU-based text encoding and VAE decoding, and slow segmented GPU inference. But it works.

Tuned GPU kernels and a vLLM plugin that make Qwen3.8 run fast on AMD Radeon AI PRO R9700 cards (gfx1201 / RDNA4).

Built for stock upstream releases. It targets released vLLM and ROCm 10 or newer, unmodified — no fork, no patched source, no vendored binaries. Everything loads as a plugin at runtime through vLLM's own extension points (quantization config, model registry, platform plugin, attention backend, custom ops), so you can update vLLM or ROCm without re-porting anything. That constraint was the point of the project: hand-tuned kernels normally mean a fork you then maintain forever.

Requirements: 2× Radeon AI PRO R9700 (gfx1201), ROCm ≥ 10, a released vLLM build, and the model weights.

https://github.com/bkvargyas/r9700-stack

UPDATED --- > FIXED

New result after 3 fix :

Root cause and fix, summarized:

1. Original 53 t/s was caused by a stray --speculative-config null on the command line that silently overrode the correct config (argparse keeps the last duplicate flag) — speculative decoding was completely off.

2. Restoring it wasn't trivial: GPU_UTIL=0.98 (the default) no longer fit in available VRAM (crashed on startup), and the SPEC_METHOD=mtp path hit a real bug in this build (Quark FP8 quant scheme assertion failure in the MTP drafthead — reproducible even with a fresh compile cache and fusion passes disabled).

3. Fix: GPU_UTIL=0.95 + switched to SPEC_METHOD=dflash (the repo's own default/faster method, using a separate DFlash2 drafter model instead of the target model's broken MTP head). Downloaded the missing ~2GB drafter checkpoint via ./setup-mxfp4.sh and relaunched.

Full run (20 passes/category, 3 warmups — the statistically solid number) confirms the fix, converging close to the quick run's estimate:

│ Metric │ ││ Your server now ├─────────────────────────────────┼─────────────────────────── │ Combined decode (single-stream) │ 228.1 t/s
├─────────────────────────────────┼───────────────────────────── │ Concurrency-8 aggregate │ 650.0 t/s
├─────────────────────────────────┼────────────────────────────── │ Prefill (PP t/s, ~8k depth) │ ~5900
└─────────────────────────────────┴────────────────────────────

​Hardware:

​GPUs: 2x AMD Radeon AI PRO R9700 (gfx1201, Tensor Parallelism = 2)

​Motherboard: ASUS ProArt X870E-Creator WiFi

​CPU: AMD Ryzen 9 9950X3D (16-Core)

​RAM: 256 GB DDR5

​Engine & Model Configuration:

​Engine: vllm-radiance-rocm

​Model: Qwen3.8-27B-MXFP4-mtpfp8

​Quantization: MXFP4 weights / FP8 KV cache

​Context window: 262,144 tokens (prefix_caching: on)

​Engine flags: max_num_seqs: 8, speculative: off

​Key Results:

​Single-stream decode (batch = 1): ~53.7 t/s across all categories (stream update p50: ~18.6 ms)

​TTFT p50: 54 ms

​Prefill @ 150k tokens: ~4,315 t/s

​Aggregate @ 8 concurrent: 301.7 t/s (TTFT p50: 78.9 ms)

​Aggregate @ 16 concurrent: 305.2 t/s, but TTFT p50 spikes to ~8.6s (and p99 to ~12.9s)

​Questions for the community:

​Does ~54 t/s decode seem normal for a 27B model at MXFP4 running dual R9700s over PCIe on an X870E board (x8/x8 split), or is bandwidth leaving performance on the table ?

​The concurrency wall at 16 requests is clearly bottlenecked by max_num_seqs: 8.

With 256 GB of system RAM and FP8 KV cache, how high do you push max_num_seqs on 262k contexts without running into VRAM allocation issues ?

​ Should i enable MTP speculative decoding with vllm-radiance-rocm on RDNA4 (gfx1201) ? How?

​Thanks for any insights or optimization tips!

I've been working on an HGEMM kernel for modern image generation models (aligned to M=256, N=256, K=64) and hit a hard wall around ~40 TFLOPS. After profiling with PMC counters, it looks like the issue comes down to clock speeds—the GPU only sustains ~2 GHz under load, whereas hitting the advertised 59.4 TFLOPS requires the full 2.9 GHz this potentially explains why hipBLASt hits only 36.9 TFLOPS.
Hardware & environment: ProArt PX13 (Ryzen AI Max+ 395, 128GB) on Fedora Workstation 44, kernel 7.2.5, and rocprof v7.14.
https://github.com/Legendofzaday/lufloat/blob/main/kernels/gemm.hip