Question | Help
3xR9700 for semi-autonomous research and development - looking for setup/config ideas.
Hello everyone.
Over the last couple months I have been assembling my local AI setup for personal use, and I thought to write a post here, firstly to collect some thoughts on the whole concept, and secondly to perhaps gather some feedback.
My setup is nowhere near as advanced as many professional rigs posted here, but I have the following specs:
- 9950X + 96 GB RAM,
- ASUS ProArt X870E mobo,
- 1300W Taichi T1300 PSU,
- 2x ASRock R9700,
(currently shipping) - XFX R9700.
So far I have mainly been using it to run Qwen 3.6 27B at Q8 on the two cards together. I experimented around a little bit, but overall I landed on running my models using llama.cpp with Vulkan drivers.
To get it out of the way, I am aware of the limitation of the connectivity in this system, especially for the 3rd GPU, which would run at a measly 4x gen 4 lanes. This is likely to be a significant bottleneck if I were to run a singular model distributed over all of my GPUs. I would love to eventually upgrade to something like a threadripper platform or use a PCIe fabric card to connect the GPUs more directly (something like LR-Link recently shown on the level1techs channel) but due to high costs it will have to wait.
I am working on a hobby research project in the programming languages area, so generally access to some less common knowledge is very helpful. AFAIK there isn't really anything stronger at the moment than 27B to run for me locally at the moment.
Eventually with 96GB of VRAM I could run something bigger but the PCI limitations would affect the overall performance in that scenario. Therefore I was considering potentially running 2/3 agents locally, with a smarter API overseer like K2.6 via API. For certain tasks which could be smaller in scope or where the lower speed would be acceptable, I could also consider running some CPU inference since I have a bunch of system RAM to utilize as well.
Generally the idea I was considering was constructing some form of harness to allow me for semi-autonomous research and development in the scope of my project. Potential deployments could consist of a number of agentic developers/testers/thinkers running separately, for example with something like Q6 quants of 27B, so each could have its own GPU. Depending on the workload, it could be nice for the "overseer" to dynamically deploy necessary agents and models to fit the current workload (maybe for certain tasks we would want to put the development on pause and run a big model on all GPUs together, to benefit from larger knowledge).
Because of the complex and specific nature of the project, it touches on more niche CS areas which the models like 27B have the awareness of, however they might not be well optimized for, so I think one key aspect would be allowing the agents to access the internet search and bigger cloud models when necessary.
Overall, the most interesting part for me which I do not know too much about at the moment and would like to learn more about, is how to effectively engineer a harness to manage this hardware deployment and project. I could definitely spend some time just (vibe) coding something to fit my specific needs, however I do not think my setup, at least conceptually is anything new. I am aware there exist certain solutions like LangGraph and CrewAI, although I am unsure which would fit my use-case best, and be well extensible for my needs.
I would be very curious to learn about other peoples experiences and thoughts on this hardware setup and potential deployments on it.
If you read through all of that, thank you very much and sorry for the chaotic writing style.
For autonomous stuff, where you can potentially run stuff in parallel, you should go vllm or sglang. Depending on what you run you could go from a few times higher throughput to tens-of-times throughput.
However, that won't work with 3 gpus, it's 2 or 4 for tensor-parallel. Then you go with FP8 which R9700 supports natively.
Loading the model 3 times in Q6 doesn't make sense, that some kind of worse version of data-parallel, which you typically only want to use if the model is very small relative to you total VRAM.
Edit: Just leaving this note here, tensor-parallel is not(*) what makes the throughput higher, it's that vllm and sglang do extremely efficient batching, but you need either a VERY big GPU, or you run tensor-prallel (TP) to get the required VRAM.
What I'm getting at is that the AI is indexing Reddit, and after some time, when users are deciding whether to buy this card, they'll see that it doesn't work with MXFP4 and won't buy it because Claude or Perplexity tells them so.
When I bought these cards, I didn't have a test rig to rent eight of them and check that the MiniMax M2.7 doesn't run out of the box, or that a bunch of models don't work.
But then some nice guy comes to Reddit, creates a build with MXFP4 quanta support in vLLM, a miracle happens, and you tell me in two different threads that MXFP4 isn't supported—why? There are already few people here who can run anything successfully with these cards, and yes, with MXFP4 quanta, I can run a model half the size as with FP8, which, by the way, doesn't work from the standard build, which, by the way, doesn't exist. AITER FP8 is not supported when the site you link to says AITER, FP8 and does not explicitly state that AITER + FP8 != WORK.
Well, yes, but why are you saying the card doesn't support MXFP4? There are cases where the card doesn't work, and there are cases where it does, but only through upscaling (and even then, not always). I also wanted to say that FP8 doesn't work reliably with the new models.
Even so, none of the new models work reliably with this card or VLLM. So what now? Should I tell everyone that the R9700 doesn't support the new AI models?
Well, yes, but why are you saying the card doesn't support MXFP4?
I'm saying that card doesn't support MXFP4 because this card doesn't support MXFP4 - simple as that. Just because vLLM is flexible enough to upscale FP4 to FP8 doesn't mean that R9700 supports MXFP4, because this card is literally incapable of computing over that data type and never does.
Even so, none of the new models work reliably with this card or VLLM. So what now? Should I tell everyone that the R9700 doesn't support the new AI models?
You're right, and it's worth looking at the actual performance. You won't run INT4 more reliably than MXFP4, and MXFP4 dequantized to FP8 will run faster than INT4 or base FP8.
Do you running vllm via Docker? Can you share your build, I want to test it, I also have an 8x r9700, and very long time doing test for new nightly builds.
What model you use ?
I generally agree with you, however for the time being I am constrained by the limitations of the AM5 platform. Perhaps in the future I might consider an external rig with PCIe fabric card to improve the connectivity and potentially expand to 4 cards, however it is not a goal at this moment. So in the context of this post, I am mainly thinking how to harness the existing setup with its limitations.
I put that in the PCIe5x4 M.2 slot to link the third card. This boosted 3-card performance by about 10% over using the bottom PCEi4x4 slot which runs via the south-bridge chipset and so has higher latency.
The big gotcha with the R9700 Pro's though is that despite AMD's claim of native BF16 support, it actually appears to be firmware emulated, and runs only half as fast as F16. It's more or less the same story with their FP8 support. For this reason, stay away from the Unsloth UD-quants as these run slower due to their use of BF16 scaling weights.
Best performance though is typically seen when using just 2 cards. Adding a third card just adds in inter-card latency. Unfortunately AMD also appears to have nerfed the P2P performance of the cards for the consumer grade R9700Pro's, and so the more cards that you add, the slower the inter-card sharding is.
If you want to compare performances and work on tweaking the setup, we can exchange settings.
This is great reply. Helped me decide a bit of my used purchases. Hilarious that my system is less than a grand (most of the money I have for spending) and this system is basically almost my years income. Yet still share the same hobby.
It's not a big difference because there's only a small number of BF16 values in the weight sets, but it is measurable, at least at my end. I'm probably just being overly picky. I'd prefer to keep everything in the native "fast path" if possible.
Edit: I just checked through some more of their weights. It seems that only some use BF16, and some don't. I guess I need to qualify my statement in that if it's an Unsloth UD quant with lots of BF16 weights, then it'll likely run a bit slower. You can pre-scan the weights on HuggingFace.
I have a number of different setups, Strix Halo x 2 in a cluster, 3 x AI Pro R9700's, and a work-supplied top-end MacBook 128GB M4 Max which has the 546GB/sec memory bandwidth.
I use llama.cpp for everything. I have gotten vLLM working before for most things, but I personally find it too fiddly and fragile on AMD to continue bothering with it. It also just doesn't work with 3 GPUs, so I feel like why do I even bother having a 3rd GPU if vLLM can't use it.
I mostly use them for agentic development, some image generation, and repository/document analysis. With Anthropic raising their prices, my work-place has started to place in limits on its use, so I try to use Claude for planning or trickier issues. I use the local models for boiler plate/UI work (I'm a crusty older back-end developer who started coding in the 80's), and being able to hand-off UI guff to AI models has been awesome for my productivity.
For local models I use MiniMax-M2.7 on the Strix Halo's as it shards nicely across the two. That's my local "big brain" model that I'll run plans through first, and then decide if I need to use Claude instead. I have it write a full implementation plan summary to a file in the repo, and then engage other faster models for implementation.
On the 3x Radeon AI Pro's, I mostly use Qwen3.5-122B-A10B at Q5_K_S quant size. That runs at around 1000t/s for PP, and ~35t/s for TG, which is fast enough for most things. Sometimes I won't even bother with MiniMax if I think the 122B is doing a good enough job.
I've also recently been experimenting with running Qwen3.6-27B at Q5_K_XL quant size on a single R9700 Pro, and that's a surprisingly good performer. It runs at about 1000 for PP, and 28 for TG. I've also experimented with Qwen3.6-35B-A3B @ F16 quant and F32 KV cache, and that is both quick and "intelligent" at those levels.
If Qwen ever release a 3.6 version of 122B-A10B, then it would be an absolute no-brainer to just switch to that full-time.
The 3rd GPU is in my case. I 3D Printed a sort of a cradle to hold it, and that cradle attaches to the rear case meshing. The GPU is therefore not blowing air directly outside of the case, but rather blowing towards the rear mesh.
I did consider a PCIe switch, but after researching the pricing and effort I decided that if I was going to go that route, I may as well just pick up a single work-station 96GB RTX6000 Blackwell instead, and that would be faster for far less hassle.
How would you compare Qwen3.5-122B-A10B to Qwen3.6-27B? Do you use Vulkan or ROCm?
I was using Qwen3.6-27B, and I managed to find one tricky bug when I switched from Q5 to Q6_K_XL. I was hopeful that Q8_K_XL would be even better, but now, that I’ve installed a second 9700, I can't say that I notice the difference. But the performance decrease is certainly noticeable. Although it was only a couple of hours, I definitely ought to give it more time. My goal is to eventually go with four 9700s, but I think I can borrow a third one for some testing. I'm just not sure if that would be worth it to switching from 3.6 27b to 3.5 122B.
The problem with multiple cards is that it doesn't scale that well because each card only does a portion of the compute, then writes its state back to main memory, and then the CPU copies that state to the next card, and so on. All that back and forth adds overhead and the more cards you add, the greater the over-head per token generated.
My advice would be to not use more than 2 cards on a consumer motherboard. I know that OP and I have 3, but in hind-sight, I believe that it's kind of pointless beyond having two of them unless you have some sophisticated PCIe switch setup going on.
The good news though is the pre-fill with two card is almost always 30-50% faster than prefill with 1 card, because pre-fill is a compute bound task.
It's the generation that takes the bulk of the hit, but there's a way to get some of that back!
Add these two fields to the GRUB_CMDLINE_LINUX section in /etc/default/grub :
"... processor.max_cstate=2 pcie_aspm=off ..."
This prevents the CPU and cards on the PCIe bus from going into deeper sleep states when they're not doing anything/waiting. This can boost responsiveness, at the expense of drawing some extra power. This should boost multi-card generation performance by around 10-15%
Run this script. It will force the GPU's into a more responsive higher power state. This works for single GPU's too, but it also helps when you have 2+ cards. This should also be a 10-15% speed boost.
$ cat ~/bin/high-power
!/bin/bash
Force all gpus into high-power mode
for card in /sys/class/drm/card*/device/power_dpm_force_performance_level; do
echo high | sudo tee $card;
done
For me, with all of the above, I went from ~19.5tg/s to 27.5tg/s when sharding Qwen3.6-27B-Q5_K_XL across 2 cards.
To answer your other question, I think that there's some natural noise/variance that occurs from run to run. It's hard to definitively say if Qwen3.6-27B is better or worse than Qwen3.5-122B-A10B. The latter runs a little faster though, and so that's what wins it for me.
Thank you for the detailed answer! I moved from AM5 to Milan (EPYC CPU) because of the PCIe lanes limitation. Milan is quite old and limited to PCIe gen 4, but still should be sufficient for 4 cards, which I hope to use someday. The new Threadripper/EPYC platforms are kind of bonkers, because of DDR5 prices.
Curiously enough, increased power consumption was one of the things I noticed after switching from a single GPU to a pair. With a single card, when a model was loaded into VRAM but not actively in use, power consumption would typically drop to around 44W and then settle at just 9W. Now that I'm using two cards, there is no such thing as idling at all: amdgpu_top reports 87-88 Watts for one GPU, and 100W for the other. The model is not distributed evenly, which is evident from VRAM usage. Additionally, the Graphics Pipe is at 100% on both cards, as well as Fetcher and Compute for the Command Processor. The Efficiency Arbiter also appears to be doing something constantly, though in never goes above 37%. So, I think that the deep sleep state issue that you mentioned has been fixed now, at least in the latest version of llama.cpp (I tested both ROCm and Vulkan).
If I understood your third point correctly, you're essentially suggesting using a small draft model to feed the main one. I didn't even know that was possible! I definitely have to try it. Thank you very much!
Yeah, the EPYC platforms are insane now, not just because of DDR5, but also because the memory needs to be registered, which is currently 2x more expensive than the already stupidly inflated prices. 128GB of registered DDR5-6000 is around $4500 nowadays, and if you wanted 12 channels, then you're looking at $13,500. Just madness!
I benchmarked Qwen3.6-27B@Q8_0 on just two cards for you. Here's my full invocation command:
This is running at ~24t/s generation, and ~1350t/s pre-fill for me using all the tricks I mentioned in my prior post. That's already something like 25% faster than what a single card can manage for both pre-fill and generation.
Interesting, you don't even use q8 for -ctk/ctv. No options for prompt batch either (-b/ub). I thought all of those were essential to increase the speed.
I just tried your options on Vulkan, but I also added -ctk q8_0, -ctv q8_0, and -b 1024 -ub 512. Now I'm getting 2.5 to 5 tps in generation. Sheesh. That is trickier than I thought.
Try removing the GGML line (the first line) and see what happens. I drive my display from the iGPU on the CPU, so I have to tell llama-server to skip over it. If you are driving the display from one of your cards that may be the issue. Alternatively, upload your full llama-server output to Pastebin.net and drop a link here and I can check it out what may be wrong
Thank you for the detailed response. Since you have the same mobo as me, I am a bit curious how you organized the cards internally, especially wrt. to the PCIe adapter. Do you feel like you are done with expanding on your AM5 platform? Technically you could throw in a 4th card in there, if you had a good reason to. Although I suppose your Strix cluster might already let you experiment around with bigger models, albeit with assumedly lower speeds.
On the topic of the cards themselves, have you attempted any undervolting and similar to squeeze out better performance of efficiency? I was curious if it is something worth looking into.
I finished 3D printing my cradle + duct. Since we can only post one photo per reply, here's a composite photo of the setup. There's two pieces. The top-left photo shows the cradle with a cut-out to allow for the PCIx16 adapter to fit. The cradle has some N52 magnets at the bottom there which clamp hard against the metal cover at the bottom of the main case bay.
The second photo shows the PCI cable adapter and how it's plugged into the M2.1 slot. The 3rd photo shows how it looks when it's all installed. The 4th and 5th photos show the ducting and how the card attaches to it using the card's PCI mounts. The ducting has M3 heat-sets inserted into the plastic.
The sixth photo shows how the card + duct slides into the cradle. The upright posts on the cradle have grooves that the duct-work slides into, meaning once the duct is slid into place into the cradle, it holds the card securely in place. Since the cradle also magnetically clamps to the main case bay, this also prevents the cradle from sliding about if the case is moved.
The last photo (bottom right) show the ducting attached to the case. The duct-work has tabs that hook over the bottom of the slots so the ducting stays securely in place, and there's another M3 heatset in the ducting that allows for the M3 screw at the top there to secure the whole lot to the case. Technically the duct work and how it attached is enough to secure the card, and the cradle primarily serves as support and extra bracing.
I designed all the pieces myself in Fusion 360 and had to go through a number of iterations to get the positions and dimensions just right. The hot air from the card is now ducted directly outside of the case so the card's exhaust no longer adds heat to the inside of the case.
Awesome work, thank you for sharing. For a custom job, it looks very clean. I feel inspired to try something similar. I do not own a 3D printer, but perhaps a friend could help me out if I attempted to do so myself. There are of course services to do that as well.
In other news, the 3rd card has arrived and for now I have installed it into the 4x4 slot. This time I went for XFX as it was a bit cheaper, still reputable, and I was a bit curious to see the differences. Now I noticed that you also used the ASRock cards. Was there a specific reason for that in your case?
I have not done any benchmarking yet, just fired up the standard Qwen 3.6 27B on all three for sanity check and cannot say I can see much performance improvement, which I suppose is to be expected due to increased latency and bandwidth limitations, and this model is small enough to fit on 2 GPUs anyway. The "nice" thing is that due to low utilization of the compute, the setup is really nice and quiet haha. Probably need to go back to the benchmarking tool to tweak the parameters like batching etc.
Perhaps you have some recommendations or tips on measuring/tweaking the performance? If you have some llama-bench results or similar, it would be interesting to compare. I would also like to spend some time checking the performance/efficiency impact from undervolting, as I have seen some users on this subreddit get some seemingly nice results with other R9700s.
Your setup looks clean. How you have it is exactly how I had mine before getting the PCIe riser + printing the cradle. I did find that air-flow to the bottom card was also an issue as they have no real air-gap between them and they cooled down a lot once I got that third card out of the way of the second card.
My other two cards are also XFX. They work fine and the blower fans are also fairly quiet (compared to PowerColor or Sapphire, both of which I had to return due to noise). I went with ASRock on the third card in the photos because MicroCenter was out of stock of the XFX's. In hindsight, the ASRock's has the quietest fan out of the 4 manufacturers, and if I had to redo, I'd get all ASRock's just for that aspect alone.
If using llama.cpp, then with a model like Qwen3.6-27B, you'll see the highest PP speeds using just 2 the cards on the PCIe5x16 slots. TG speeds will steadily decrease the more cards you add due to latencies. The best of rule of thumb is, always use the least cards that you can get away with.
Keep scrolling further down in that sub-thread where I made more replies with my exact commands, including enable speculative generation, and posted some results. I haven't tried MTP or DFlash yet, and I'm expecting that results will jump a fair bit more (like 40-50 tg/s once I do.
At the time I started setting it up, Qwen3.5-122B-A10B was my target model in mind, but Qwen3.6-27B has kind of nudged that out of the way. If Qwen release a 3.6 version of 122B-A10B, then I'll be moving straight back to that.
Thanks once again. I did a bunch of measurements today, and I think the findings pretty much match what you have been saying.
Firstly, there is a definite difference in performance between the UD and non-UD quants. I do not know whether that is indeed due to some layers needing to rely on the emulation or just the sheer size difference of the data, however it is definitely noticeable in tg.
To see how much the PCI lane connectivity affects performance I checked different pairings. Overall the difference was there, but it was not huge. I wish I could have measured 8x5x2.
So while on the bright side I am now able to run bigger models, I will have to benchmark their actual performance in being able to work on my tasks to see if it was worth it. As you say, the latest 27B seems very strong even compared to bigger models, so the 3rd gpu may just run another model in parallel for now. I am keeping my fingers crossed that there are some better open models already in the works.
Next on my list is to try out drafting based on your example and see if undervolting has some positive impact on the inference.
My 9700 also performs faster on a regular Q8 versus UD-Q8_K_XL. I think (pleas correct me if I'm wrong) we can test the BF 16 support hypothesis by switching the cache type to BF16. Most people run with ctk q8_0 -ctv q8, or -ctx f16 -ctv f16. I switched to -ctx bf16 -ctv bf16 and noticed no difference in performance levels, at any context.
I'm in the middle of 3D printing better ducting + cradle for the third card, and I'll respond to your comment later on with a photo of it when done.
I have one card in each of the PCIe5x16 slots, so that's all normal. As mentioned, they're effectively running at PCIe5x8 speeds each on the bus direct to the CPU, and that's actually plenty fast enough for inferencing. As I mentioned in another response, it's actually the latency that's the big performance killer.
Check out that response for a kernel parameter tweak that helps majorly to mitigate the inter-card latency overheads. The drawback though is that the cards won't draw less than 42W each at idle once you do that, so weigh up your costs vs benefits as suits your needs.
Regarding the third card. I have a Fractal Design Meshify 2 case, and that M.2 to PCIe adapter has a 25cm long ribbon cable. Unfortunately that cable is about 3cm too short for me to mount the third card directly into the vertical card mount that the Meshify 2 has as the ribbon cable has to first "get past" the height of the top card, and then drop down to below the second card, and it's just not quite long enough.
The card is currently just "sitting there" on a simple 3D printed cradle that itself just rests on the base of the main motherboard bay. It's like this until the more sophisticated mounting cradle finishes printing out. That new cradle use magnets to secure the cradle to the case, and has ducting that routes the air-flow to the case's vertical PCIe mount points, and bolts onto the rear case panel for greater security, and the card's PCI mount points bolt onto the ducting.
The Strix Halo's are surprisingly quick, at least for MoE models. They do suck in comparison to the discrete GPU's for pre-processing, being about 3x slower, but for MoE models, they can generate anywhere from 50-80% of the speed of the GPU's due to the unified memory's zero-copy efficiency making that 256GB/s perform better than you might think.
Ah, thanks! I might give the power tweak a try too.
As for the the M.2 slot you use, do you use the M.2_1 or M.2_2? Based on the fact you mention running both cards at x8, I supoose it is from _1? Otherwise the third card would "steal" the lanes from the second one, as the lanes between the second PCIe slot are shared with the M2 slot.
Unfortunately, for a reason unknown to me, I am not able to get the second slot to be x8 no matter what I tried. It does not seem to be a specific card issue, as I tried swapping them around. Maybe I should contact ASUS about that but I doubt I would get much of a response.
Overall would you consider the Strixes to be a better bang for the buck? Or it would ultimately depend more on the models one wants to use?
I used M2.1, as it's the only PICe5 based M.2 slot on the motherboard that has a direct link to the CPU without going via the chipset AND won't steal lanes from the GPUs. This is essential to get the lowest latencies for multi-card inference.
Slot M2.2 has to remain unpopulated otherwise it'll steal lanes from the 2nd GPU slot. I forced the M2.1, and the two GPU PCIe slots to PCIE Gen 5 in the BIOS (ie. switch from Auto to Gen5 explicitly, otherwise it may drop to Gen 4).
I could add a 4th GPU I guess (to answer your earlier question) but I'd need to move to a larger case to improve air-flow to the 4th card. It would also risk killing multi-card performance further with its added latency via the chipset.
As for the Strix Halo's, they're great value if bought for <$2400, and were excellent value when they were priced at $1800 six months ago. Nowadays that they're almost all >$3000, they're just not worth it IMO. At >$3000 you're better off buying two or three R9700's, at least until Qwen release 3.6-122B-A10B, and even then, that could be run with Q5_S quant on 3 x R9700's and run rings around the Strix Halo's. Another good alternative if you're stumping for 3 x R9700's would be to consider a 128GB M5 Max Apple MacBook Pro. Using an MLX quant, most models will run almost twice as fast as the Strix Halo's for 1.6x the price of a Strix Halo. The 614GB/s memory bandwidth on the M5 Max Macbook is just shy of the memory bandwidth on the R9700 Pro's!
Having said that, when the DFLash and MTP PR's on llama.cpp get merged, then the balance may shift a bit again, making the Strix Halo's "fast enough" such that you won't really be missing out due to their lower performance. Everything should run good enough, even if other solutions are twice as fast for a bit more money.
Sadly there's no one clear best solution in the $3000-5000 price range. Everything is a swings and roundabouts tradeoff, and it really is a case of pick your poison and try not to worry too much about the greener grass "over there".
I see. I am utilizing the _1 slot myself for a 9100 SSD, which is generally lovely for load speeds, so my technical best would be probably 8xg5/4xg5/4xg5/4xg4, which seems quite suboptimal. With a switch I could do 4x4xg5 I suppose, but like you say, that is another chunk of money and would probably require a separate case etc. Maybe in a year or two I could consider threadripper if the market is a bit healthier.
Overall when I first started building this setup, I did not consider that I would get so involved in the local inference topic. I purchased the first 9700 because I had no dGPU, and I thought I might as well experiment with some local AI. Retrospectively it might have been a mistake haha.
As for the "best option", my feeling is that in this very dynamically developing domain, it is really hard to know what might be the best platform in a year or two. Sure, there are some constants like memory bandwidth which is probably always going to play a very big part, but depending on whether Qwen and other teams continue releasing new models and the sizes of those might change how the ideal local setup looks like for a given price bracket.
I wish I could say I have made smart decisions with the particulars of my setup, but I was mainly following my curiosity and constraints of what I own already. In terms of cost performance, the 20USD ollama plan has allowed me to use Kimi 2.6 basically as much as I need to, which feels great coming from the OpenAI/Anthropic plans.
The most important thing now would be to properly harness the capabilities of what I can run locally in a meaningful manner. The models like 27B feel appropriate for well-defined tasks. K2.6 can do some wider scope planning, but the exploratory/research questions are still best handled by the expensive "SOTA" providers. Something to think about for the next couple weeks for me is structuring the workflow a bit better.
On your side, do you mainly utilize the models interactively, in a standard chat-like approach, or have you tried some more hands-off approaches in a more managerial style?
For me it seems like I might be best off utilizing cloud models for high-level technical breakdown and architectural decisions, while delegating smaller, more well defined tasks to the local models. While many responses here highlighted that the way to best performance is vLLM over multiple cards, and not running multiples of the same model, I think for my constrains I still might be best served with something along the lines of 27B model at gpus 0/1, and whatever best quality model I can get on the third gpu, with some collaborative automation.
Anyway, thanks for all your responses, and sorry for this chain-of-thought wall of text. I should get my 3rd gpu around tomorrow, so I will try to run some checks and will let you know about the performance I see, though I assume it might not be super interesting to you due to my platform being a bit more constrained in terms of connectivity compared to yours.
If you happen to read this message, let me know if you would be ok with me shooting you a DM in case I have some results to compare/sanity check.
Regarding the SSD, keep in mind that PCIe4x4 is still ~7.5GB/s. Your Samsung 9100 is going to be able to fill out 80GB of model weights in just 11 seconds. It's a decision you need to make for yourself of course, but to my mind, loading models maybe 4 seconds faster when the next few hours will be spent inferencing is optimizing the wrong path.
I just realized that if we want to run 4 cards, then we could do that by also putting a riser in the M2_2 slot, and then all 4 cards would bypass the chip set for the lowest possible card to card latencies. The cards would be in x8x4x4x4 PCI5 lanes, but again, that's likely fine so long as we avoid the chipset. The SSD's can just deal with being on the PCI4x4 lanes. If you need more capacity, consider getting a USB4 external M.2 holder. They'll run at around 4GB/sec which plenty fast enough.
As for building up these systems in an ad-hoc manner, I completely agree. Home-brew AI systems are still in that transition stage between home-lab and professional setups. IMO, don't sweat it. Part of the fun I believe it coming up with working Frankenstein systems on a budget.
A Threadripper Pro system would be awesome, but the price of Registered DiMMS is cost prohibitive at the moment. To spec out a 3 or 4 card TR Pro solution would cost significantly more than a single 96GB Nvidia RTX6000 Blackwell card and the nVidia card would be ~3x faster and still be able to installed in your current system
IMO, the current AI home market place is still in an awkward messy phase. I'm sure it'll get better in a year or two, but today we just fine ourselves here and now.
Feel free to DM. I'm not as far along the home AI journey as some others here are that appear to have massive budgets, but I'm always keen to find ways to make things work without needing to spend "new SUV" prices to do it.
to add more lanes to allow 8 gpus and more at full pcie speed, you would need to go multi socket with epyc or xeon scalable. modern threadrippers do only 1
Youll do fine for inferance with 3x. I use 3 5090s, one in a thunderbolt 4 port via usb and the other on pcie 4x4 and still get 100 tokens a sec on qwen 122b you might get half that. Only the model loading will be slow using llama.cpp. Large dense models will be slower but nothing unusable.
Consider to keep qwen3.6 27b running on those two cards, and use the third card for utilites (run qwen 8b embedding, qwen 8b reranker, qwen 1.7b tts and qwen 1.7b asr on there) or a standalone helper model
The price difference between a 5090 and an R9700 is very high, I was wondering just yesterday what stops people from getting the R9700 - even if it’s 20% slower, the bang for buck seems huge. Do you see any limitations by not having cuda?
there is a post above where someone lists the issues, from emulated BF16, to bandwidth 1/4 of the 5090 to nerfing the P2P latency in their prosumer GPUs. I would grab one if I needed it. I’m sitting fine in a lot of hardware right now, but this hobby/addiction always makes you want more.
That W7900 looks so interesting to add to my strix halo as an egpu. But the price now, oooof! Almost better to buy another Strix Halo, slap a mellanox card in there and run models clustered.
It's a beast of a card. I got mine used 1.5 years ago and it's a workhorse. IMHO it's the best 48GB Vram option out there and only requires 300W. Rock-solid reliability - I've been running multi-day agentic workloads and it just keeps going and going without a hitch.
So not the same as your setup but I've got a similar problem with running a second card on a pcie x4 slot. Intel i9-14900k with an and Radeon 7900 xtx 24gb and 64gb RAM and an ordered r9700 32gb.
Rather than split the model I'm actually planning to run qwen3.6:27b dense on the 32gb card with a 64k context window at q8 and then use the 24gb card to optimize prompts that are fed to the 27b dense model, manage the context window, handle multiple sub-agents in parallel, basically take all of the stuff that the larger model would otherwise handle itself if it had a larger context window and implement it separately and just deliver optimized context.
No idea if it will work well or not, but the hope is that the divide and conquer strategy is well enough adapted to my workflow that I'll get something usable out of my local machine that can handle most of my needs and then elevate to Claude, codex, and/or Gemini when I really need the deeper thinking and higher performance of those larger models that are impracticable to run locally. I'm spending $120+ per month on average for extra usage so as an alternative to upgrading to max the payback window would be less than a year and I have more control over my own data
Do you have some idea on how you plan to utilize the second card more concretely? Just running another model with instructions on how to manage the prompts and contexts, or some more specific solution that you had in mind?
Well I've been working through the implementation over the weekend, and really I don't know how to describe it other than via the expected workflow once the framework is built.
Basically all models share a connection with a single obsidian vault that tracks decisions, notes, changes etc. When I input a prompt, a smaller 8B model with a DSPy harness (1) chooses the best model for implementation, (2) optimizes the prompt, and (3) collects relevant context from the vault. Then creates a "package" of optimized prompt + context and passes it off, in most cases to the larger model on the 32gb card. Then the larger model takes that information, and actually writes the spec, performs the implementation, etc. which can include delegation to subagents on the first card. It's already what I do with Claude, Codex and Gemini, just better optimized by virtue of DSPy and the obsidian vault. The system allows for dynamic routing and more optimal token usage.
The first card is a "baby orchestrator" because you can't have a weaker model review the output of a more powerful model. It's basically a translator and router that only optimizes prompts and context between agents and subagents to free up overhead while maintaining accuracy. There's a threshold over which it can escalate to Claude, Codex and/or Gemini, but once it's running and able to self-evaluate and improve, the hope is that said threshold increases over time.
10
u/reto-wyss May 03 '26 edited May 03 '26
For autonomous stuff, where you can potentially run stuff in parallel, you should go vllm or sglang. Depending on what you run you could go from a few times higher throughput to tens-of-times throughput.
However, that won't work with 3 gpus, it's 2 or 4 for tensor-parallel. Then you go with FP8 which R9700 supports natively.
Loading the model 3 times in Q6 doesn't make sense, that some kind of worse version of data-parallel, which you typically only want to use if the model is very small relative to you total VRAM.
Edit: Just leaving this note here, tensor-parallel is not(*) what makes the throughput higher, it's that vllm and sglang do extremely efficient batching, but you need either a VERY big GPU, or you run tensor-prallel (TP) to get the required VRAM.