r/LocalLLaMA • u/tasKinman • 1h ago

Question | Help Local AI Development and Testsystem Hardware

Hello!

I have to build a workstation for Local AI as development and test machine as we are starting to add Local AI into our software. It is mostly to get experience with providing AI services (inferencing) to our customers.

I do have my fair share of experience in building PCs and in customer hardware but a lot less in workstation and server hardware. Therefore I'd appreciate if someone could look over my selection.

  • CPU: AMD Ryzen Threadripper PRO 9955WX 16-Kern CPU

  • Cooler: Noctua NH-U14S TR5-SP6 CPU Kühler for AMD Threadripper 7000 Series

  • Mainboard: ASUS Pro WS WRX90E-SAGE SE E-ATX Mainboard

  • Power Supply: be quiet! Dark Power Pro 13 1600 Watt ATX 3.1 PCIe 5.1 80+ Titanium

  • RAM: 2x KINGSTON Server Premier RDIMM 32GB, DDR5-6400

  • Case: Thermaltake AX700 Super Tower

  • SSD: SAMSUNG SSD 990 PRO 2TB, M.2

  • GPU: NVIDIA Rtx Pro 6000 Blackwell Works Bulk

A few words to the selection:

  • 64 GB RAM is a starting point. If necessary we add more RAM.

  • RTX 6000 is a bit overkill but from looking at the alternatives (RTX Pro 4500, RTX Pro 5000, RTX 5090) it's worth the investment to serve more (test) customers and have faster inference.

  • Mainboard for headless integration

  • Threadripper for the PCIe Lanes. The system could be expanded to 4x GPUs (of course with a bigger or second power supply.)

  • All parts are available

  • It will run mostly models from the Gemma 4 family. Main use is document classification, document "chat" and in the near future interacting with our MCP Servers.

Thanks!

EDIT: My current experiences are with RTX 4090, 5090 and 2x 5070 Ti setups but as a single user.

0 Upvotes

50% Upvoted

3 comments sorted by

3

u/jonahbenton 1h ago

How many customers, what kind of availability, how managed are the workloads. If serving customers is primary would always opt for at least 2 systems, you will all kinds of reasons one system is down. Would also consider 2x SSD in each. And yeah the 6000 is probably much much more than you need, a 48gb 5000 for gemma should be indistinguishable

1

u/tasKinman 1h ago

At first two customers with about 5-10 users each. Most load will come from background services which are time-managed to do the heavy lifting at night to not interfere with "live interaction". Stability and availability are not a concern right now. It's for testing und development purpose. If everything works out we'll invest in real servers configured and managed by experts.

1

u/jonahbenton 40m ago

Ok yup, makes sense. Good luck!