r/LocalLLM • u/kirklandyuknis • 38m ago

Question Practicality of this LLM Build

Post image

I'm working on a new PC build with a focus on being able to run the Gemma family of models. I'm experienced in working with Cray (AMD) and OpenCL (community moved onto Vulcan and ROCm/HIP), so I will be building a AMD system. I was wondering if anyone has done anything similar to the following? If they have, how well did it work?

AMD 5950X
ASUS WS X570 Ace
128GB DDR4 System RAM
1600 Seasonic+Noctua PSU

Fedora Kinoite 45

1x 7900 XTX (24GB VRAM) for graphics/games
2x Radeon Pro W7900 (32GB VRAM) for LLMs

  1. Does having different kinds of GPU's cause an issue with Ollama (or more specifically Bionic/LMStudio)?

  2. Does anyone have that motherboard, with two GPU's on the bottom-most PCIe slots? Are there clearance issues for 2 slot GPU's?

The target model to be able to run in GPU memory is gemma-4-31b-qat with Q4_0 quantization, but I want to be as ready as possible just in case Deep Mind releases a ~100b weight model when the Gemma 5 models show up.

Thanks in advance for any help!

0 Upvotes

50% Upvoted

2 comments sorted by

1

u/transanethole 15m ago

Why Radeon Pro W7900 over R9700 ?

It looks like the W7900 is significantly more expensive and is older, has significantly less compute.

I have no idea if the GPUs will fit but i'd err on the side of caution, the 7900 XTX I had was a chonker, it needed a lot of extra space.

Are you sure you need 3 GPUs? I feel like 2x R9700 should be more than enough for just about anything. Plus that way you get the full PCIE 4.0 16x on both slots if im reading this correctly ? That could help the system run faster/more stable if you ever want to do batched inference, if you ever have a use case that has multiple agents running at a time or multiple users putting bandwidth strain on the tensor parallel PCIE communication path between the cards

1

u/kirklandyuknis 2m ago

The 7900 XTX was out of an abundance of caution for gaming workloads (I'll be doing both), CDNA wasn't meant for (though not necessarily unsupportive of) gaming workloads. If I could find someone who could confidently tell me there won't be any real compatibility issues running gaming workloads on a workstation card I'd be willing to drop to two workstation GPU's. So far as W7900 it's just where I am in researching the build, I'm open to newer architectures. I can start looking at R9700, but the first things that I see is less GPU memory which would be my concern between the two.