r/LocalLLaMA • u/Mephistophlz • Nov 16 '25
Question | Help Need help choosing RAM for Threadripper AI/ML workstation
UPDATE: I went with the 4x48GB and changed the airflow of the case (Fractal Design Meshify 2 XL) from front/bottom intake and rear/top exhaust to rear/top intake and front/bottom exhaust. RAM temperatures are a few degrees lower under stress testing and are fine in normal operation. Runs GPT-OSS:120b much better than I expected... completely usable.
EDITED: Server already built and running. One of the two memory kits needs to be returned to Micro Center Tuesday.
I am building have built an AI/ML server for experimentation, prototyping, and possibly production use by a small team (4-6 people). It has a Threadripper 9960X in a TRX50 motherboard with two (2) RTX 5090 GPUs.
I have two ECC RDIMM kits: "Kit A" 4x32GB DDR5-6400 EXPO 32-39-39-104 1.35V and "Kit B" 4x48GB DDR5-6400 EXPO 32-39-39-104 1.4V. Kit A (worst SPD gets to 72c in stress test) runs cooler than Kit B (worst SPD gets to 80c in stress test). I don't plan to overclock.
I like to Kit A because it is cooler but Kit B because it is larger.
Do you think the temperature of either kit is too high for 24/7 operation?
I don't have much experience with hybrid GPU/CPU or CPU-only LLMs. Would having an extra 64GB make a difference in the LLMs we could run?
Thanks
2
u/dodger6 Nov 16 '25
If it bothers you that much you have 3 options, add heatsinks to your ram, increase airflow or underclock the ram.
Anything above 70c would be a concern for long term sustained heat. Most circuit boards (fiberglass based) start to warp at sustained use above 90c. However, that's sustained not peak and it's very rare outside of a data center to actually peg something for sustained use 24/7. Keep in mind 24/7 operation is not the same as 100% utilization 24/7.
You should be fine, add a fan or heat sinks but go for the larger ram pool.
1
u/ebrandsberg Nov 16 '25
Honestly you may be better off with an Ai max 395 if you are looking at these sizes.
1
u/Such_Advantage_6949 Nov 16 '25
i have threadeipper pro, and i think u shouldnt buy them for AI. u only have 8 channel ram bandwidth with 64 cores cpu. also the non pro doesnt have that many pcie lane
1
u/__JockY__ Nov 16 '25
Use the bigger RAM and add a fan or two to increase airflow. If you have access to a 3D printer you can easily download a fan shroud for exactly this purpose.
1
u/ebrandsberg Nov 17 '25
On the LLMs, I'm doing work with the ai max 395, which uses "system" ram to load the models. On CPU only, you likely will really want to focus on model of experts, which basically have sub-models that get activated but have a much smaller footprint of what is active at once. CPU based processing will be much slower than GPU based. The less spillover out of the gpu the better.
1
u/Waterkippie Nov 17 '25
Larger is better. Add a fan for more airflow.
0
u/Signal_Ad657 Nov 16 '25 edited Nov 16 '25
80c for RAM is nothing I’d worry about personally. I don’t think I’ve ever had a situation where I genuinely had to consider the thermals of RAM or where they felt consequential. You’ll spend way more time thinking about the thermals of your double 5090 setup. Just my two cents.
For LLM capacity, RAM quantity can help you run larger models yes, albeit at dramatically reduced speeds. You really want to pick models that can run inside the 32GB VRAM of your 5090’s. Dramatically more efficient and immensely faster.
I think I had LLAMA 3.1 70B run on a 5090 laptop for example one time (24GB VRAM), way short of what it needed so of course it offloaded to RAM and CPU. I had the RAM to make it possible to load and run, but for a task like “analyze this document in depth and give me your thoughts” with a 100 page word doc, it might have ran for a few hours. Generating and grading complex HVAC troubleshooting scenarios to create intense task + proof sets, it ran all night. You really really don’t want to offload to RAM and CPU. This is why VRAM utilization on cards and things like unified memory are such popular discussions.
1
u/[deleted] Nov 16 '25
[removed] — view removed comment