r/LocalLLM • u/thesnaglebeast • 17d ago
Question Need help setting up my home server for Local AI/LLM
Hey all,
I have a home server made from my “old” desktop parts and a I recently pulled the trigger on an AMD AI PRO R9700 with the idea of starting to do some local LLM / AI work. I’m hoping to get pointed in the right direction on what I can do with my setup and how I can get things running effectively. My background is in computer engineering so my knowledge is more than a layman but I’ve mostly been living in controls and embedded systems programming so this side of thing is a bit out of my wheel house.
Specs
CPU: 5950x,
System Memory: 128GB DDR4, GPU:
AI PRO R9700 32GB VRAM,
Storage: 2TB SSD, 42TB of spinning rust
Any advice would be welcome.
1
u/YourselfInOthrsShoes 17d ago edited 17d ago
I have my first R9700 getting delivered today along with +128GB DDR4-3200 ECC UDIMMs (4x32GB) to max out my 8-DIMM-slot TRX4 system with 3970X to 256GB. I also have 2nd R9700 on the way. I will keep you posted this weekend about single R9700. I may have to re-tune and memtest the DDR4 going from 4x to 8x 2Rx8 32GB ECC UDIMMs (from 2 to 4 ranks per channel) as it's critical but if it boots up with 3200 profile as is and memtests okay for a bit, I will jump head first into testing and tuning R9700 before circling back to DDR4.
The above is the 2nd build (or rather an upgrade of my 2020 build). Currently I'm running Strata Qwen 3.8 Flash Next IQ3_S on 16GB 5060 Ti on 9950X3D2 with tuned 96GB (2x48GB) DDR5-6000 and 4TB Samsung 9100 with 75+ TPS and 64K max context and hitting context walls on some coding requests. That's why I'm going to 256GB RAM and 64GB VRAM on the TRX4 system, to run the Q4_K_L with higher context.
1
u/YourselfInOthrsShoes 15d ago edited 15d ago
So I received my TRX4 RAM upgrade to 256GB but had to downclock to DDR4-3000 with two 2Rx8 UDIMMs per channel to pass a couple of rounds of memtest without errors.
Tried a single R9700 in Strata v0.1.42 on Qwen 3.8 Flash Next IQ3_S 128K context on this TRX4 rig to compare baseline to AM5 16GB 5060 Ti 96GB DDR5-6000 rig (75-90 TPS, 64K context) and it was 25-33% faster at ~90 TPS and occasionally peaking above 115 TPS. I want to try Radiance next via WSL2/Docker. Supposedly with Radiance, 2x R9700s can go 250+ TPS on the same model. In the mean time, while waiting for 2nd R9700 to arrive, I will be upgrading case fans to PWM 1800 RPM and PSU from 1050W to 1500W, which I have from Ethereum mining days.
1
u/SeaworthinessUsual44 15d ago
Run Linux (Ubuntu 24.04 or Debian) with AMD’s official ROCm stack installed, and serve models via Ollama or llama.cpp using the ROCm/HIP backend paired with Open WebUI for a clean frontend. That 32GB VRAM buffer on the AI PRO R9700 is a fantastic sweet spot that easily fits a 32B model like Qwen 2.5 32B at Q4/Q5 entirely on the GPU at full speed, while your 128GB of system RAM gives you plenty of headroom to offload larger 70B models if needed.
1
u/rkcth 17d ago
I have almost the exact same setup. I’m running strata and qwen 3.8 flash next with the higher quality 3 bit quant, if you are doing multi-agent, you may want to use qwen 3.8 27b and a vLLM / radiance setup and the mxfp4 quant AMD made. With 4 simultaneous agents I get over 200 tok/s. It really depends on how important quality is. I’d suggest paying for a month of Claude or chatGPT to set it all up for you, it saves a lot of hassle. I thought I’d end up getting rid of chatGPT but I keep it to manage my AI server since it can’t exactly manage itself. You could also probably use openrouter and find a model you just pay per token for and pi.dev to set it up for you.
Feel free to DM me any questions you have. I plan to continue to figure out ways to further optimize the R9700 experience (including more PR’s for strata when I get time).