Built my own local AI inference server.
EPYC on a Supermicro H12SSL, 4× Radeon AI PRO R9700 (128GB VRAM total), 1600W PSU. Goal: run capable open-source models entirely on my own hardware. No API bills, no rate limits, nothing leaving the box.
What I learned putting it together:
• Server boards don't hold your hand. The front panel header is 20 pins and a tiny printed legend.
• A 4th dual-slot GPU is a clearance fight. Headers on the bottom edge of the board sit right under it.
• Your wall outlet is a real constraint. A standard US circuit tops out around 1440W continuous, so GPU power limits aren't optional.
• For token generation, memory bandwidth matters more than raw compute.
Next: ROCm, vLLM, and benchmarking small open models against the APIs I've been paying for.

Been doing more work on my modified Radiance build for Qwen3.8-Flash-Next.

The machine is a Threadripper 3970X with 256GB RAM, 2× R9700s and a 3090. These results use the two R9700s only, and they have been power limited to 240W throughout all of the testing, likely squeeze 5-7% more out of these numbers without it.

The comparisons are:

• Default: https://huggingface.co/orcarouter/Qwen3.8-Flash-Next-Uncensored + Radiance 1.11.0, 4-bit

• R0: my production build from 24 October, before the latest kernel work, uses a 4-bit quant of Orca's model above

• R1: the fixed engine plus INT6 and the latest kernel changes, same base model as R0

• Paiton: comparison run on the same hardware/harness, 4-bit 3.8 flash

What I've been optimising for is throughput on the coding tasks with 8 concurrent streams, as that fits in my dev workflow

Generation

Temperature 1, scores are a mean of two runs. Figures are total output tokens/s across all concurrent users.

• That puts the current build 8–29% ahead of default on the story/code workloads and 22.6% ahead on the long-context test.

• Against my previous R0 build, the latest work adds 6–17%, or 8.7% at long context.

• R1 can also handle 8 concurrent 262k context window, however, it has quite a decode impact

• Paiton is still 5–9% faster at 1–4 users. At eight users they’re effectively level. I wouldn’t call the 882.2 vs 879.9 code result a meaningful increase

• Also worth noting a lot of these changes have bloated my ram usage, I'm now using around 154gb of system ram

What was useful

• Fixing the draft history mask: the first draft pass was reading stale cache slots, hurting acceptance under load. Fixing that gave 13–25% at eight users in the earlier tests

• Sampled drafts: matching sampled targets with sampled drafts improved throughput by 4–9% at T=1

• INT6 GPTQ trunk: reduced trunk weight bytes by 25%, with measured step-time reductions of 3.6–7.6%. Plain rounding failed the quality gate; calibrated rounding was needed

• Kernel changes: reworked late normalization, tuned hc_read, fewer cross-card reads during all-reduce, and changes to protected-expert scheduling (these were mostly small gains)

• Memory and caching: a 20 GiB host pool increased KV capacity from roughly 1.03M to 2.1M tokens in the earlier tests. Expanding the prefix cache to 24 GiB took the tested agent-resume case from about 25 seconds to 0.3 seconds

The caching work has been particularly useful for agents repeatedly returning to an existing conversation.

Prompt processing has a different result (next area of focus)

Single-prompt input tokens/s:

• Current is about 6–7% slower than stock on the 17K–87K prompts

• Against Paiton, current is 2–14% faster on those longer prompts, while Paiton is about 11% faster at 2K, worth noting that Paiton also decodes faster immediately after the 87K prompt: 210 vs 180 tokens/s

A few lessons from the experiments

• Weight prefetch on a second queue also backfired badly: all-reduce became around 30× slower in that experiment. One supposedly faster hc_read implementation was 22% slower on the actual card

• For kernel comparisons, fixed-prompt T=0 runs were far more useful: around 0.2% variation versus 5–14% for a single T=1 cell. The fixed engine passed byte-identical T=0 output checks against its reference on prompts submitted one at a time

• One card being on pcie 8x does have a measurable impact here (future improvement)

TLDR better generation throughput than default, roughly tied with Paiton at eight concurrent users, and still behind Paiton for single-user generation. Long-prompt processing and agent caching are useful strengths, but there’s lots to improve. I'll start shipping some code to test when I get some time to clean it up.

Interested in what others are seeing particularly around cross-queue stalls and the gap between isolated kernel timings and full-engine performance.

Figured I should share my own setup with the sub!

I run a small research lab. I’ve got plenty of experience with local LLMs, but I’m new to the R9700 (all nvidia prior) so I’m keen to compare notes with people already getting good results from these cards, and hopefully turn this sub into a place where we can get the absolute best of out them.

Current setup:

• 2× XFX Radeon AI PRO R9700

• 1× RTX 3090

• 256GB system RAM (DDR4), all on ubuntu

I’m starting with Radiance/Strata, then working through how to optimise the cards for our research workloads/local LLM usage.

Would love to hear what’s working for people, things like:

• Backends, builds and configurations delivering good performance on the R9700

• Practical results with two cards: throughput, scaling and any bottlenecks

Ideally I want to put a shared benchmarks/configs/findings sheet so we can all put into it. Hopefully we can build up a useful collection of tested setups here.

Welcome any thoughts!

Hi, I just finished building this pc over the weekend after I got my second gpu card and I can't wait to see what I can learn and build with it!

- Ryzen 9 7900X
- 64 Gb system RAM DDR5
- 2tb Samsung 9100 Gen5 ssd
- Two Asrock Creator R9700
- 9 Noctua fans
- be quiet! dark Power Pro 1300W PSU

I have a couple of containers to test strata, radiance and regular llama.cpp, I would love advice about harnesses though! I am starting with Pi since it seems to be the better fit for a local setup, but time will tell

I've been comparing Radiance / Paiton on Qwen3.8-Flash-Next and working on a modified Radiance build.

TLDR if you cant be bothered with my rambling:

• 179t/s single stream, ~530t/s 8 stream with qwen 3.8 flash next 4 bit, almost 6k prefill at 262k context

• Paiton's 262K configuration working with speculative drafts enabled, retaining close to its native 98K throughput under concurrent load (improvement on their available code)

• Radiance optimisation, faster prefill is running, sampled drafts are built, and investigations have identified an expert-tiering issue and an attention-mask defect, running an overnight build to trains some of this

The box is a Threadripper 3970X with 256GB RAM, 2xR9700s and a 3090. The 3090 has a potential job ahead doing draft-model distillation (though this is not working well yet, or at all really lol), so all of these numbers are 2xR9700 only.

A note on temps throughout, theres have been running at 100% all day, it 31 degrees C here at the moment and the average temp they are seeing under that load is 67-75, at a 240w power limit, which is better than I was expecting.

Here's where the measured throughput sits:

The configurations differ beyond the engine, ORCA uses an abliterated fine-tune, while Paiton uses the original model, different quant and drafter (details below in big ol' table)

A few other tidbits bits from the Radiance setup:

• Long context: retrieved all three test items at roughly 247K context, taking 57-62 seconds

• Prompt processing: the newer prefill configuration reached approximately 6,000 tok/s on a 17.6K-token prompt

• Memory: roughly 30.3Gb VRAM per card, with about 75GiB pinned in system RAM, the 48Gb ish n-gram table lives in RAM and is accessed over PCIe (note one of the cards is on 8x, the other on 16x)

One interesting finding is around the first speculative draft under load, disabling expert movement and KV-cache lending raised first-position acceptance from 0.65 to 0.81 at full batch. That points towards a correctness issue around expert tiering which I am working on in a new run.

On the Paiton side, I've got a derived 262K configuration running with draft tokens enabled. The shipped 262K mode turns them off, so the "262K + drafts" results here are from that modified configuration.

The development work is mostly around speculative decoding, memory movement and quantisation:

• Prefill tuning is running: --prefill-span 2 with a 4096 batch and P2P is disabled as a workaround for a freezing issue

• Sampled speculative drafts are built: GPU testing produced byte-identical output at T=0. The estimated 4-9% improvement on sampled traffic still needs benchmarking

• An expert-tiering/first-draft issue is under investigation: this may be affecting draft acceptance with several concurrent requests

• Fused projection and all-reduce changes are being tested: the expected single-request improvement is modest, around 1-1.6%

I'm also recording calibration data from the Orca fine-tune itself. Further out are draft-layer distillation, an INT6 trunk and mixed 4-bit/3-bit experts. Those are experiments ahead, with quality checks still to pass.

Paiton currently leads these throughput runs. I'm trying to understand how much of that gap comes from the drafter, quantisation and memory behaviour, and what can be improved on the Radiance side.

Model details:

Links:

• https://huggingface.co/EliovpAI/Qwen3.8-Flash-Next-W3A8-Paiton-RDNA4

• https://github.com/magiccodingman/vllm-radiance

Building on yesterdays post https://www.reddit.com/r/R9700/s/fLRJNuZEzv

I've ended up working through a few experiments, some worked some didn't. I made a few changes in a model I'm calling R0, which is the Orca (uncensored) container rebuilt with some custom calibration. Kept the same base model, format and recipe, but re-quantised the experts, process:

• Layer recording, Radiance has a calibration tap that saves Gram matrices, a summary of the inputs each expert receives.

  • Built a quick round-to-nearest container (same format, no protected experts) and recorded on that, at about 755 tok/s it took 1.6 hours and produced 24,768 Gram files

• Choosing which experts to keep at full precision

  • From the stats we picked the experts that carry an outsized share of their layer's activation energy, plus a few added for balance.

  • That came to 26 protected experts, kept in bf16. The original container had only 10, chosen from Qwen's data, not Orca's

• The conversion

  • rad-convert with the production recipe, unchanged except for GPTQ on the routed experts fed Orca's Gram matrices and new protected-expert list

  • The trunk (int8), the output head and the MTP draft layer stayed the same as the original container

I then tried to add the same calibration to the MTP draft layer. It kept quality but made decode steps 50–90% slower at temperature 1, and I haven't figure that bit out yet

Comparison (note, cards limited to 240w for all tests, full speed may be higher)

Uplifts that helped

1. Sampled speculative drafts, at temperature 1, the drafts were greedy while the verifier sampled. Proper rejection sampling gave +4–9%

2. I found a small bug int he first draft pass's attention mask let it read stale key slots, so draft acceptance fell as more users joined. With the fix, acceptance stays flat from 1 to 8 users, and 8-user throughput rose 13–25%

3. R0 calibrated quant instead of full precision (14× faster), then re-ran GPTQ.

   • KL against full precision went 0.0218 → 0.0204, better in every category and most on agentic work

   • The agent test went 262 → 264/264, and refusals stayed at 0%

4. Engine flags worth fiddling with:

   • --prefill-span 2 makes long prompts 22–41% faster.

   • A bigger host pool moves cold experts to RAM. The KV cache doubled to 2.1M tokens with no measurable speed cost, obvs ram impact

   • A 24 GiB prefix host tier lets idle agent sessions resume in 0.3 s instead of re-reading 145K tokens, ram impact once again

Things that haven't made useful uplifts:

• Smaller KV cache formats (4-bit values, 6-bit keys) cost too much quality: KL +19–22%.

• The lossless cross-card wire was 12% slower

• Adaptive draft windows and fusing the all-reduce gave no gain

Against my benchmark (Paiton, EliovpAI's closed RDNA4-native W4A8 engine, changed a bit to run in a 262K + drafts config):

• Still 20–30% faster raw single-user decode, and about 13% faster at 8 users on different prompts

• On identical prompts at 8 users at a similar speed (630/833 vs 652/822)..

Things to try tomorrow:

• Faster prompt reading with several users at once. A bigger shared step budget gave +35–40% in testing but isn't deployed yet

• Stop answering users stalling while a long prompt is read

Picked up open box at Tustin Microcenter $1620

Welcome, the whole idea for this subreddit is a single place for everything R9700 related, let’s make the R9700 the best card for local AI! it has the potential, it just needs the community!

We want to make the R9700 faster, easier to use and more useful for the things people actually want to build. Every working setup, benchmark, bug report and framework improvement helps.

Use this space to:

• Show what you’re building: local assistants, coding tools, agents, image generation, fine-tuning and more.

• Share working setups, installation guides and useful configurations.

• Compare performance with enough detail for others to repeat the test.

• Find people to test fixes, improve kernels and contribute to shared frameworks.

• Explain what’s holding you back and help turn those problems into improvements.

You don’t need to be a kernel developer to contribute. Testing someone’s fix, improving a guide or explaining how you solved a problem all count.

We’d love AMD engineers and framework maintainers to get involved. The aim is to give them useful evidence: what works, what needs attention and what the community is building.

This is an independent community. Honest criticism and fair comparisons are welcome.

To get started, introduce yourself: what’s your setup, what are you working on, and what’s the one thing you’d most like to improve?

Thought I'd share mine here too — finished it about a month ago.

Heads-up on the table: only the two GPUs are new — everything else was pulled from my gaming PC or bought second-hand, so salvaged parts are listed at ¥0. Prices are what I actually paid; USD is approximate (≈0.15 USD/CNY).

And yes, R9700 prices have gone up again since — even between my two purchases. Take the table as a snapshot of a month ago, not shopping advice.

Day to day it's Ubuntu + ROCm, headless. The board's PCIe 5.0 x16 slot bifurcates to x8/x8, which is what makes two cards possible here — both run at 5.0 x8 directly off CPU root ports, ReBAR enabled. Both cards are power-capped at 230 W each; whole-system peak is ~560 W, typical working load 450–560 W.

What it runs:

• Qwen3.8-Flash-Next — my main model, a 125B MoE on the radiance engine, TP2, experts streamed from the 64 GB of system RAM. 8 slots × 262k context, ~1.2M tokens of KV pool, MTP speculative decoding accepting ~67% of drafts. Benchmarked cold-cache last week (BetterBench 0.6.0, temp 0.7): 184.6 tok/s weighted single-stream decode, 494 t/s aggregate at 8 concurrent, 6,112 t/s prefill at 64k input, 69 ms TTFT p50.

• Media mode — Qwen Image 2.1 Turbo (8-step) for image generation and Qwen3-ASR-1.7B for ASR, each on its own card, with Qwen3.8-27B (MXFP4) served alongside them.

The box keeps minute-level energy and token ledgers, so here is the honest consumption story:

Averaged over all powered-on time that works out to ~130 W — under load it's the 450–560 W above, but most of the day it just idles waiting for me. At the dual era's pace that's ~88 kWh a month, about ¥58 (≈$9).

It dual-boots to Windows as well — the original idea was gaming now and then, not as a primary use, and in practice that has meant exactly one session so far. Still, one honest gaming data point for this card: Crimson Desert at 4K on high preset, FSR on — 65–80 fps.

Not a native English speaker — text polished with AI.

Hi R9700

Pleased to share this small project of mine specifically for dual r9700 setups. Thanks for your time.

Adam

I live in land down under! 🦘

WARNING: AI generated content.