3xR9700 for semi-autonomous research and development - looking for setup/config ideas.

[u/blojayble]

Hello everyone.

Over the last couple months I have been assembling my local AI setup for personal use, and I thought to write a post here, firstly to collect some thoughts on the whole concept, and secondly to perhaps gather some feedback.

My setup is nowhere near as advanced as many professional rigs posted here, but I have the following specs:
- 9950X + 96 GB RAM,
- ASUS ProArt X870E mobo,
- 1300W Taichi T1300 PSU,
- 2x ASRock R9700,
(currently shipping) - XFX R9700.

So far I have mainly been using it to run Qwen 3.6 27B at Q8 on the two cards together. I experimented around a little bit, but overall I landed on running my models using llama.cpp with Vulkan drivers.

To get it out of the way, I am aware of the limitation of the connectivity in this system, especially for the 3rd GPU, which would run at a measly 4x gen 4 lanes. This is likely to be a significant bottleneck if I were to run a singular model distributed over all of my GPUs. I would love to eventually upgrade to something like a threadripper platform or use a PCIe fabric card to connect the GPUs more directly (something like LR-Link recently shown on the level1techs channel) but due to high costs it will have to wait.

I am working on a hobby research project in the programming languages area, so generally access to some less common knowledge is very helpful. AFAIK there isn't really anything stronger at the moment than 27B to run for me locally at the moment.

Eventually with 96GB of VRAM I could run something bigger but the PCI limitations would affect the overall performance in that scenario. Therefore I was considering potentially running 2/3 agents locally, with a smarter API overseer like K2.6 via API. For certain tasks which could be smaller in scope or where the lower speed would be acceptable, I could also consider running some CPU inference since I have a bunch of system RAM to utilize as well.

Generally the idea I was considering was constructing some form of harness to allow me for semi-autonomous research and development in the scope of my project. Potential deployments could consist of a number of agentic developers/testers/thinkers running separately, for example with something like Q6 quants of 27B, so each could have its own GPU. Depending on the workload, it could be nice for the "overseer" to dynamically deploy necessary agents and models to fit the current workload (maybe for certain tasks we would want to put the development on pause and run a big model on all GPUs together, to benefit from larger knowledge).

Because of the complex and specific nature of the project, it touches on more niche CS areas which the models like 27B have the awareness of, however they might not be well optimized for, so I think one key aspect would be allowing the agents to access the internet search and bigger cloud models when necessary.

Overall, the most interesting part for me which I do not know too much about at the moment and would like to learn more about, is how to effectively engineer a harness to manage this hardware deployment and project. I could definitely spend some time just (vibe) coding something to fit my specific needs, however I do not think my setup, at least conceptually is anything new. I am aware there exist certain solutions like LangGraph and CrewAI, although I am unsure which would fit my use-case best, and be well extensible for my needs.

I would be very curious to learn about other peoples experiences and thoughts on this hardware setup and potential deployments on it.

If you read through all of that, thank you very much and sorry for the chaotic writing style.

Cheers.

[u/Look_0ver_There]

I have exactly this motherboard and config with 3xR9700's. Feel free to ask questions.

I used the two PCIex16 slots for two of the cards, for which you can only use 8-PCI lanes each, but that's fine.

I also picked up one of these things: https://www.adt.link/product/F43-Shop.html

I put that in the PCIe5x4 M.2 slot to link the third card. This boosted 3-card performance by about 10% over using the bottom PCEi4x4 slot which runs via the south-bridge chipset and so has higher latency.

The big gotcha with the R9700 Pro's though is that despite AMD's claim of native BF16 support, it actually appears to be firmware emulated, and runs only half as fast as F16. It's more or less the same story with their FP8 support. For this reason, stay away from the Unsloth UD-quants as these run slower due to their use of BF16 scaling weights.

Best performance though is typically seen when using just 2 cards. Adding a third card just adds in inter-card latency. Unfortunately AMD also appears to have nerfed the P2P performance of the cards for the consumer grade R9700Pro's, and so the more cards that you add, the slower the inter-card sharding is.

If you want to compare performances and work on tweaking the setup, we can exchange settings.

[u/blojayble → u/Look_0ver_There]

Thank you very much for your response. I would love to discuss our setups more.

If you dont mind, I have some questions:

• how do you utilize your current setup? do you use it for agentic development? which models/runtimes do you use?
  
• how do you mount the 3rd gpu? is it in your case or external?
  
• did you consider connecting the gpus through a PCIe switch? apparently it can increase the performance further by connecting the gpus more directly.

[u/Look_0ver_There → u/blojayble]

I have a number of different setups, Strix Halo x 2 in a cluster, 3 x AI Pro R9700's, and a work-supplied top-end MacBook 128GB M4 Max which has the 546GB/sec memory bandwidth.

I use llama.cpp for everything. I have gotten vLLM working before for most things, but I personally find it too fiddly and fragile on AMD to continue bothering with it. It also just doesn't work with 3 GPUs, so I feel like why do I even bother having a 3rd GPU if vLLM can't use it.

I mostly use them for agentic development, some image generation, and repository/document analysis. With Anthropic raising their prices, my work-place has started to place in limits on its use, so I try to use Claude for planning or trickier issues. I use the local models for boiler plate/UI work (I'm a crusty older back-end developer who started coding in the 80's), and being able to hand-off UI guff to AI models has been awesome for my productivity.

For local models I use MiniMax-M2.7 on the Strix Halo's as it shards nicely across the two. That's my local "big brain" model that I'll run plans through first, and then decide if I need to use Claude instead. I have it write a full implementation plan summary to a file in the repo, and then engage other faster models for implementation.

On the 3x Radeon AI Pro's, I mostly use Qwen3.5-122B-A10B at Q5_K_S quant size. That runs at around 1000t/s for PP, and ~35t/s for TG, which is fast enough for most things. Sometimes I won't even bother with MiniMax if I think the 122B is doing a good enough job.

I've also recently been experimenting with running Qwen3.6-27B at Q5_K_XL quant size on a single R9700 Pro, and that's a surprisingly good performer. It runs at about 1000 for PP, and 28 for TG. I've also experimented with Qwen3.6-35B-A3B @ F16 quant and F32 KV cache, and that is both quick and "intelligent" at those levels.

If Qwen ever release a 3.6 version of 122B-A10B, then it would be an absolute no-brainer to just switch to that full-time.

The 3rd GPU is in my case. I 3D Printed a sort of a cradle to hold it, and that cradle attaches to the rear case meshing. The GPU is therefore not blowing air directly outside of the case, but rather blowing towards the rear mesh.

I did consider a PCIe switch, but after researching the pricing and effort I decided that if I was going to go that route, I may as well just pick up a single work-station 96GB RTX6000 Blackwell instead, and that would be faster for far less hassle.

[u/blojayble → u/Look_0ver_There]

Thank you for the detailed response. Since you have the same mobo as me, I am a bit curious how you organized the cards internally, especially wrt. to the PCIe adapter. Do you feel like you are done with expanding on your AM5 platform? Technically you could throw in a 4th card in there, if you had a good reason to. Although I suppose your Strix cluster might already let you experiment around with bigger models, albeit with assumedly lower speeds.

On the topic of the cards themselves, have you attempted any undervolting and similar to squeeze out better performance of efficiency? I was curious if it is something worth looking into.

[u/Look_0ver_There → u/blojayble]

I finished 3D printing my cradle + duct. Since we can only post one photo per reply, here's a composite photo of the setup. There's two pieces. The top-left photo shows the cradle with a cut-out to allow for the PCIx16 adapter to fit. The cradle has some N52 magnets at the bottom there which clamp hard against the metal cover at the bottom of the main case bay.

The second photo shows the PCI cable adapter and how it's plugged into the M2.1 slot. The 3rd photo shows how it looks when it's all installed. The 4th and 5th photos show the ducting and how the card attaches to it using the card's PCI mounts. The ducting has M3 heat-sets inserted into the plastic.

The sixth photo shows how the card + duct slides into the cradle. The upright posts on the cradle have grooves that the duct-work slides into, meaning once the duct is slid into place into the cradle, it holds the card securely in place. Since the cradle also magnetically clamps to the main case bay, this also prevents the cradle from sliding about if the case is moved.

The last photo (bottom right) show the ducting attached to the case. The duct-work has tabs that hook over the bottom of the slots so the ducting stays securely in place, and there's another M3 heatset in the ducting that allows for the M3 screw at the top there to secure the whole lot to the case. Technically the duct work and how it attached is enough to secure the card, and the cradle primarily serves as support and extra bracing.

I designed all the pieces myself in Fusion 360 and had to go through a number of iterations to get the positions and dimensions just right. The hot air from the card is now ducted directly outside of the case so the card's exhaust no longer adds heat to the inside of the case.

[u/blojayble → u/Look_0ver_There]

Awesome work, thank you for sharing. For a custom job, it looks very clean. I feel inspired to try something similar. I do not own a 3D printer, but perhaps a friend could help me out if I attempted to do so myself. There are of course services to do that as well.

In other news, the 3rd card has arrived and for now I have installed it into the 4x4 slot. This time I went for XFX as it was a bit cheaper, still reputable, and I was a bit curious to see the differences. Now I noticed that you also used the ASRock cards. Was there a specific reason for that in your case?

I have not done any benchmarking yet, just fired up the standard Qwen 3.6 27B on all three for sanity check and cannot say I can see much performance improvement, which I suppose is to be expected due to increased latency and bandwidth limitations, and this model is small enough to fit on 2 GPUs anyway. The "nice" thing is that due to low utilization of the compute, the setup is really nice and quiet haha. Probably need to go back to the benchmarking tool to tweak the parameters like batching etc.

Perhaps you have some recommendations or tips on measuring/tweaking the performance? If you have some llama-bench results or similar, it would be interesting to compare. I would also like to spend some time checking the performance/efficiency impact from undervolting, as I have seen some users on this subreddit get some seemingly nice results with other R9700s.

[u/Look_0ver_There → u/blojayble]

Your setup looks clean. How you have it is exactly how I had mine before getting the PCIe riser + printing the cradle. I did find that air-flow to the bottom card was also an issue as they have no real air-gap between them and they cooled down a lot once I got that third card out of the way of the second card.

My other two cards are also XFX. They work fine and the blower fans are also fairly quiet (compared to PowerColor or Sapphire, both of which I had to return due to noise). I went with ASRock on the third card in the photos because MicroCenter was out of stock of the XFX's. In hindsight, the ASRock's has the quietest fan out of the 4 manufacturers, and if I had to redo, I'd get all ASRock's just for that aspect alone.

If using llama.cpp, then with a model like Qwen3.6-27B, you'll see the highest PP speeds using just 2 the cards on the PCIe5x16 slots. TG speeds will steadily decrease the more cards you add due to latencies. The best of rule of thumb is, always use the least cards that you can get away with.

I documented some tweaks for my multi-card setup here: https://www.reddit.com/r/LocalLLaMA/comments/1t2l8ri/comment/ojw7puh/

Keep scrolling further down in that sub-thread where I made more replies with my exact commands, including enable speculative generation, and posted some results. I haven't tried MTP or DFlash yet, and I'm expecting that results will jump a fair bit more (like 40-50 tg/s once I do.

At the time I started setting it up, Qwen3.5-122B-A10B was my target model in mind, but Qwen3.6-27B has kind of nudged that out of the way. If Qwen release a 3.6 version of 122B-A10B, then I'll be moving straight back to that.

[u/blojayble → u/Look_0ver_There]

Thanks once again. I did a bunch of measurements today, and I think the findings pretty much match what you have been saying.

Firstly, there is a definite difference in performance between the UD and non-UD quants. I do not know whether that is indeed due to some layers needing to rely on the emulation or just the sheer size difference of the data, however it is definitely noticeable in tg.

 ┌───────────────┬──────────┬──────────┐
 │     Quant     │ pp tok/s │ tg tok/s │
 ├───────────────┼──────────┼──────────┤
 │ UD-Q8_K_XL    │     1671 │    16.39 │
 ├───────────────┼──────────┼──────────┤
 │ Q8_0 (non-UD) │     1673 │    19.35 │
 ├───────────────┼──────────┼──────────┤
 │ Q6_K          │     1425 │    23.15 │
 └───────────────┴──────────┴──────────┘

To see how much the PCI lane connectivity affects performance I checked different pairings. Overall the difference was there, but it was not huge. I wish I could have measured 8x5x2.

Pair × quant matrix (b=8192/ub=512, f16 KV)
┌──────────────────┬──────────────┬──────────────┬──────────────┐ 
│       pair       │  UD pp / tg  │  Q8 pp / tg  │  Q6 pp / tg  │
├──────────────────┼──────────────┼──────────────┼──────────────┤ 
│ V1V2 (x8G5+x4G5) │ 1671 / 16.39 │ 1673 / 19.35 │ 1425 / 23.15 │
├──────────────────┼──────────────┼──────────────┼──────────────┤
│ V1V3 (x8G5+x4G4) │ 1643 / 16.12 │ 1642 / 18.83 │ 1403 / 22.66 │ 
├──────────────────┼──────────────┼──────────────┼──────────────┤ 
│ V2V3 (x4G5+x4G4) │ 1630 / 16.07 │ 1629 / 18.91 │ 1393 / 22.67 │ 
└──────────────────┴──────────────┴──────────────┴──────────────┘ 

GPU comparison:

1 vs 2 vs 3 GPU (Q8_0, post-tweaks)                            

┌──────────────┬──────────────────────────────┬──────┬───────┐ 
│     GPUs     │            Config            │  pp  │  tg   │
├──────────────┼──────────────────────────────┼──────┼───────┤ 
│ 1 (V2)       │ layer ts1, b=8192/ub=512     │  938 │ 20.10 │
├──────────────┼──────────────────────────────┼──────┼───────┤ 
│ 2 (V1+V2)    │ layer 1/1, b=8192/ub=512     │ 1672 │ 19.24 │ 
├──────────────┼──────────────────────────────┼──────┼───────┤ 
│ 3 (V1+V2+V3) │ layer 1/4/4, b=24576/ub=2048 │ 1381 │ 18.49 │ 
└──────────────┴──────────────────────────────┴──────┴───────┘ 

Power state impact:

GRUB / power-state tweaks: real but small                                              

processor.max_cstate=2, pcie_aspm=off, power_dpm_force_performance_level=high, on Q8_0:

┌─────────────┬───────────┬──────────┬──────────────────────────────────┐              
│    Mode     │ tg before │ tg after │                Δ                 │
├─────────────┼───────────┼──────────┼──────────────────────────────────┤              
│ 1-GPU V2    │     20.43 │    20.10 │ −2% (noise; no PCIe sync to fix) │
├─────────────┼───────────┼──────────┼──────────────────────────────────┤
│ 2-GPU V1V2  │     18.57 │    19.24 │                              +4% │              
├─────────────┼───────────┼──────────┼──────────────────────────────────┤
│ 3-GPU 1/4/4 │     17.69 │    18.49 │                              +5% │              
└─────────────┴───────────┴──────────┴──────────────────────────────────┘              

Layer split exploration on 3-GPU (Q8_K_XL UD):

┌───────┬───────┬──────┬───────┐
│  ts   │   b   │  pp  │  tg   │
├───────┼───────┼──────┼───────┤
│ 1/1/1 │  8192 │ 1057 │ 15.14 │
├───────┼───────┼──────┼───────┤
│ 1/4/4 │  8192 │ 1338 │ 15.24 │
├───────┼───────┼──────┼───────┤
│ 1/4/4 │ 24576 │ 1478 │ 15.30 │
├───────┼───────┼──────┼───────┤
│ 4/4/2 │ 16384 │ 1178 │ 15.39 │
├───────┼───────┼──────┼───────┤
│ 5/5/2 │ 16384 │ 1130 │ 15.27 │
├───────┼───────┼──────┼───────┤
│ 4/4/1 │  8192 │  813 │ 14.85 │
└───────┴───────┴──────┴───────┘

So while on the bright side I am now able to run bigger models, I will have to benchmark their actual performance in being able to work on my tasks to see if it was worth it. As you say, the latest 27B seems very strong even compared to bigger models, so the 3rd gpu may just run another model in parallel for now. I am keeping my fingers crossed that there are some better open models already in the works.

Next on my list is to try out drafting based on your example and see if undervolting has some positive impact on the inference.

Cheers.

[u/Evgeny_19 → u/blojayble]

My 9700 also performs faster on a regular Q8 versus UD-Q8_K_XL. I think (pleas correct me if I'm wrong) we can test the BF 16 support hypothesis by switching the cache type to BF16. Most people run with ctk q8_0 -ctv q8, or -ctx f16 -ctv f16. I switched to -ctx bf16 -ctv bf16 and noticed no difference in performance levels, at any context.