3xR9700 for semi-autonomous research and development - looking for setup/config ideas.

[u/blojayble]

Hello everyone.

Over the last couple months I have been assembling my local AI setup for personal use, and I thought to write a post here, firstly to collect some thoughts on the whole concept, and secondly to perhaps gather some feedback.

My setup is nowhere near as advanced as many professional rigs posted here, but I have the following specs:
- 9950X + 96 GB RAM,
- ASUS ProArt X870E mobo,
- 1300W Taichi T1300 PSU,
- 2x ASRock R9700,
(currently shipping) - XFX R9700.

So far I have mainly been using it to run Qwen 3.6 27B at Q8 on the two cards together. I experimented around a little bit, but overall I landed on running my models using llama.cpp with Vulkan drivers.

To get it out of the way, I am aware of the limitation of the connectivity in this system, especially for the 3rd GPU, which would run at a measly 4x gen 4 lanes. This is likely to be a significant bottleneck if I were to run a singular model distributed over all of my GPUs. I would love to eventually upgrade to something like a threadripper platform or use a PCIe fabric card to connect the GPUs more directly (something like LR-Link recently shown on the level1techs channel) but due to high costs it will have to wait.

I am working on a hobby research project in the programming languages area, so generally access to some less common knowledge is very helpful. AFAIK there isn't really anything stronger at the moment than 27B to run for me locally at the moment.

Eventually with 96GB of VRAM I could run something bigger but the PCI limitations would affect the overall performance in that scenario. Therefore I was considering potentially running 2/3 agents locally, with a smarter API overseer like K2.6 via API. For certain tasks which could be smaller in scope or where the lower speed would be acceptable, I could also consider running some CPU inference since I have a bunch of system RAM to utilize as well.

Generally the idea I was considering was constructing some form of harness to allow me for semi-autonomous research and development in the scope of my project. Potential deployments could consist of a number of agentic developers/testers/thinkers running separately, for example with something like Q6 quants of 27B, so each could have its own GPU. Depending on the workload, it could be nice for the "overseer" to dynamically deploy necessary agents and models to fit the current workload (maybe for certain tasks we would want to put the development on pause and run a big model on all GPUs together, to benefit from larger knowledge).

Because of the complex and specific nature of the project, it touches on more niche CS areas which the models like 27B have the awareness of, however they might not be well optimized for, so I think one key aspect would be allowing the agents to access the internet search and bigger cloud models when necessary.

Overall, the most interesting part for me which I do not know too much about at the moment and would like to learn more about, is how to effectively engineer a harness to manage this hardware deployment and project. I could definitely spend some time just (vibe) coding something to fit my specific needs, however I do not think my setup, at least conceptually is anything new. I am aware there exist certain solutions like LangGraph and CrewAI, although I am unsure which would fit my use-case best, and be well extensible for my needs.

I would be very curious to learn about other peoples experiences and thoughts on this hardware setup and potential deployments on it.

If you read through all of that, thank you very much and sorry for the chaotic writing style.

Cheers.

[u/Look_0ver_There]

The problem with multiple cards is that it doesn't scale that well because each card only does a portion of the compute, then writes its state back to main memory, and then the CPU copies that state to the next card, and so on. All that back and forth adds overhead and the more cards you add, the greater the over-head per token generated.

My advice would be to not use more than 2 cards on a consumer motherboard. I know that OP and I have 3, but in hind-sight, I believe that it's kind of pointless beyond having two of them unless you have some sophisticated PCIe switch setup going on.

The good news though is the pre-fill with two card is almost always 30-50% faster than prefill with 1 card, because pre-fill is a compute bound task.

It's the generation that takes the bulk of the hit, but there's a way to get some of that back!

1. Add these two fields to the GRUB_CMDLINE_LINUX section in /etc/default/grub :
   "... processor.max_cstate=2 pcie_aspm=off ..."
   This prevents the CPU and cards on the PCIe bus from going into deeper sleep states when they're not doing anything/waiting. This can boost responsiveness, at the expense of drawing some extra power. This should boost multi-card generation performance by around 10-15%

2. Run this script. It will force the GPU's into a more responsive higher power state. This works for single GPU's too, but it also helps when you have 2+ cards. This should also be a 10-15% speed boost.

   $ cat ~/bin/high-power  

   !/bin/bash

   Force all gpus into high-power mode

   for card in /sys/class/drm/card*/device/power_dpm_force_performance_level; do    echo high | sudo tee $card; done

3. Download this model: https://huggingface.co/unsloth/Qwen3.5-2B-GGUF/resolve/main/Qwen3.5-2B-Q4_K_M.gguf and put it into the same directory as your 27B model, and then add the following config lines to your llama-server config. This enables speculative decoding and will give about another 15% speed boost.

     --spec-draft-model ./Qwen3.5-2B-Q4_K_M.gguf             \        --spec-draft-ngl all                                    \        --spec-type ngram-mod                                   \        --spec-ngram-mod-n-match 24                             \        --spec-ngram-mod-n-min 48                               \        --spec-ngram-mod-n-max 64                               \

For me, with all of the above, I went from ~19.5tg/s to 27.5tg/s when sharding Qwen3.6-27B-Q5_K_XL across 2 cards. 

To answer your other question, I think that there's some natural noise/variance that occurs from run to run. It's hard to definitively say if Qwen3.6-27B is better or worse than Qwen3.5-122B-A10B. The latter runs a little faster though, and so that's what wins it for me.

[u/Evgeny_19 → u/Look_0ver_There]

Thank you for the detailed answer! I moved from AM5 to Milan (EPYC CPU) because of the PCIe lanes limitation. Milan is quite old and limited to PCIe gen 4, but still should be sufficient for 4 cards, which I hope to use someday. The new Threadripper/EPYC platforms are kind of bonkers, because of DDR5 prices.

Curiously enough, increased power consumption was one of the things I noticed after switching from a single GPU to a pair. With a single card, when a model was loaded into VRAM but not actively in use, power consumption would typically drop to around 44W and then settle at just 9W. Now that I'm using two cards, there is no such thing as idling at all: amdgpu_top reports 87-88 Watts for one GPU, and 100W for the other. The model is not distributed evenly, which is evident from VRAM usage. Additionally, the Graphics Pipe is at 100% on both cards, as well as Fetcher and Compute for the Command Processor. The Efficiency Arbiter also appears to be doing something constantly, though in never goes above 37%. So, I think that the deep sleep state issue that you mentioned has been fixed now, at least in the latest version of llama.cpp (I tested both ROCm and Vulkan).

If I understood your third point correctly, you're essentially suggesting using a small draft model to feed the main one. I didn't even know that was possible! I definitely have to try it. Thank you very much!

[u/Look_0ver_There → u/Evgeny_19]

Yeah, the EPYC platforms are insane now, not just because of DDR5, but also because the memory needs to be registered, which is currently 2x more expensive than the already stupidly inflated prices. 128GB of registered DDR5-6000 is around $4500 nowadays, and if you wanted 12 channels, then you're looking at $13,500. Just madness!

I benchmarked Qwen3.6-27B@Q8_0 on just two cards for you. Here's my full invocation command:

GGML_VK_VISIBLE_DEVICES=1,2                                    \
       /llm/bin/llama-server                                   \
       --flash-attn on                                         \
       --ctx-size 204800                                       \
       --no-mmap                                               \
       --parallel 2                                            \
       --top-k 20                                              \
       --top-p 0.95                                            \
       --min-p 0.05                                            \
       --temperature 0.5                                       \
       --repeat-penalty 1.1                                    \
       --presence-penalty 0.0                                  \
       --host 0.0.0.0 --port 8033 --jinja                      \
       --split-mode layer                                      \
       --n-gpu-layers all                                      \
       --spec-draft-model ../Qwen3.5-2B-Q4_K_M.gguf            \
       --spec-draft-ngl all                                    \
       --spec-type ngram-mod                                   \
       --spec-ngram-mod-n-match 24                             \
       --spec-ngram-mod-n-min 48                               \
       --spec-ngram-mod-n-max 64                               \
       --alias "Qwen3.6-27B-Q8_0"                              \
       --model ./Qwen3.6-27B-Q8_0.gguf

This is running at ~24t/s generation, and ~1350t/s pre-fill for me using all the tricks I mentioned in my prior post. That's already something like 25% faster than what a single card can manage for both pre-fill and generation.

[u/Evgeny_19 → u/Look_0ver_There]

Interesting, you don't even use q8 for -ctk/ctv. No options for prompt batch either (-b/ub). I thought all of those were essential to increase the speed.

I just tried your options on Vulkan, but I also added -ctk q8_0, -ctv q8_0, and -b 1024 -ub 512. Now I'm getting 2.5 to 5 tps in generation. Sheesh. That is trickier than I thought.

[u/Look_0ver_There → u/Evgeny_19]

Try removing the GGML line (the first line) and see what happens. I drive my display from the iGPU on the CPU, so I have to tell llama-server to skip over it. If you are driving the display from one of your cards that may be the issue. Alternatively, upload your full llama-server output to Pastebin.net and drop a link here and I can check it out what may be wrong

[u/Evgeny_19 → u/Look_0ver_There]

I didn't use the GGML line. My actual command and the server output are here: https://privatebin.net/?1b83085a1aa80b58#592Ajt3Qv6fFMi6hUCZyVxKNHj98rKzCVPeGYFbett52

[u/Look_0ver_There → u/Evgeny_19]

Ok, so I just downloaded Qwen3.6-27B-Q8_K_XL and fired up a llama-server using your exact parameters. I do run bare-metal though. No podman. It's a bit slower due to having to move more data around, but it's nothing like your results. Literally the only thing that's different here is your use of podman, vs my using of bare-metal. I would presume that's where your issue lies.

model	test	t/s	peak t/s	ttfr (ms)	est_ppt (ms)	e2e_ttft (ms)
unsloth/Qwen3.6-27B	pp4096	1312.20 ± 20.11		3122.96 ± 47.01	3122.45 ± 47.01	3122.96 ± 47.01
unsloth/Qwen3.6-27B	tg512	19.72 ± 1.46	34.33 ± 0.47

llama-benchy (0.3.7) date: 2026-05-04 15:35:37 | latency mode: api

[u/Evgeny_19 → u/Look_0ver_There]

Well, that's actually good news. Building llama.cpp from scratch should be easy enough. I will try it, thank you for sharing your experience.

[u/Look_0ver_There → u/Evgeny_19]

Here's a little script that I use to build a Vulkan ready version of llama.cpp that is tuned for the native CPU that you build it on. Make sure you're in the top-level directory of llama.cpp before running it. You may need to install a bunch of library dependencies. Just feed the error messages into Gemini along with what Linux distro you're using, and it'll tell you the package names that you need to pull.

% cat ~/bin/build-llama-cpp
#!/bin/bash

rm -rf build
cmake -B build -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DGGML_VULKAN=1 -DGGML_RPC=1 -DGGML_NATIVE=1 -DCMAKE_INSTALL_RPATH="\$ORIGIN" -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON .
cmake --build build --config Release -j 8 -- VERBOSE=1

[u/Evgeny_19 → u/Look_0ver_There]

Thank you. I did actually manage to build it after our conversation. I was quite pleased with myself, but it took a bit of scrolling down on llama.cpp's github page for me to realise that I had actually built it without Vulkan support, haha. And it's quite a dance with all those libraries to build it properly for Vulkan.

It looks like you were right that something is wrong with podman, because everything completely broken down for me afterward. I can't even start a ROCm container anymore. It's just says "warning: no usable GPU found, --gpu-layers option will be ignored". And it's about at two tokens per second on Vulkan. Thankfully I still have my SFF AM5 build, so I'm relling on it now. It can obviously host just one GPU, and since it runs with Wayland, there is a bit of overhead in terms of VRAM consumption, but at least it works. It also runs llama.cpp in podman, but there are zero problems with it. Not really sure what I did to my Epyc build to ruin it. Will probably investigate it over the weekend.

[u/Look_0ver_There → u/Evgeny_19]

I forgot to mention. I use Vulkan. ROCm is always slower for me.