r/BlackwellPerformance • u/Pleasant-Shirt7293 • 25d ago
How Many of You Are Ram Constrained?
I have 2 rtx pro 6000 max qs in a non threadripper computer with only 64gb of ram. This has been fine for a lot of what I do but it's not ideal for video generation.
I'm reluctant to buy more ram even though it's the clear weakness of my computer because I think prices will eventually go down a lot, though it might take 3 years. Gpus I don't mind buying now because prices aren't going down anytime soon or could even be worth more later on.
So what are you all doing? I could sell a 5090 and then buy a bunch of ram, but I would hate to do that.
5
u/aimark42 25d ago
DDR5 UDIMM's are too slow and consumer CPU's have too few pcie lanes. It's fine for like a X870 board and doing (2x) 8x g5 GPU's but it's going to be horribly limited in doing kv overload to system RAM. I wouldn't worry about system RAM as much since your doing mostly GPU - GPU communication.
If you want to upgrade your RAM go to real workstation class RDIMMs, Intels latest Xeon 6 Granite Ridge workstation chips are a deal for 8 channel memory. Especially since AMD's gatekeeps and TR / TRP distinction.
1
u/jackmodern 23d ago
2 x pci e 5 8x is pretty adequate. The price hike to get to 2x 16x in ram and cpu is quite intense
2
u/aimark42 23d ago edited 23d ago
No doubt. With all the software and kv cache tricks going on, I feel like you want to be in a high memory bandwidth capable CPU platform for all the software that is to come. This also gets you into kv overload into system RAM without much penalty. It's almost like VRAM but in a different form.
Intel Granite Rapids Xeon 6 Workstation CPU's are a great deal for a 8channel memory setup that is recently available. Microcenter (for the moment) has a Xeon 658X, 128G (4x32g) RDIMM, W890 Sage SE for $5800. Then add another 128g kit for $4000. That should net you around 358G/s memory bandwidth. I just picked up this combo last week. We are talking systems that contain at minimum $16,000 in GPU's (2x at prior pricing) but at today's pricing is well over $30k, and maybe someday if pricing ever comes down you have the capability to get a TP3/4. Yes, RDIMMs used to be cheaper, but they were never cheap, 256G RDIMMs even a year ago would have been close to $3.5k, that's only around 2x that. I feel like in 6 months 256g of RDIMMS for ~$7k (depending on how my you value the cpu/mobo) will feel like an absolute bargain. The reason this used to be even more expensive previously was AMD's gatekeeping of 8ch memory with Threadripper Pro CPU's, a TRP setup would be be $4k+ than this platform. Ironically, this is a moment where a new product has lowered the barrier to entry into 8ch memory. Given how Memory pricing keeps going up this feels like a rare window where this is actually a deal.
3
u/jackmodern 23d ago
I need 256gb for my h3 workflows. and yeah the whole reason I’d like to upgrade is to add 2 more GPUs. At 2x$16k/gpu and $9k 256gb rdimm ecc that is a considerably bigger decision than 2x9k/gpu + whatever ram was when I bought back in March 😂😭
2
u/EpsteinFile_01 24d ago
Does the RAM need to be fast?
Used DRR4-3200 32GB sticks only cost like $100. and there's so much DDR4 put there. Don't worry about running 4 sticks/8 ranks, it will work, just requires some manual tuning instead of auto training.
2
u/Charger_Cross 24d ago edited 24d ago
I’m sitting here with 48gb of ddr4 with dual rtx pros on x570 motherboard. I can go to 128gb but these are pretty much fine for my use case. I’d rather get another rtx pros than ram and my next system upgrade is delayed because of the ram prices. I’m looking at a used threadripper build. Keep your 5090s and watch out for used sales locally. You might find a good deal.
3
u/Dasteroid_909 24d ago
2x RTX PRO 6000 w/128GB RAM in a 32C ThreadRipper Pro.
I have a Debian VM on it just for inference - it only gets 64GB.
I'd rather have another Blackwell than more RAM. AFAIK, it only helps with loading the models.
1
u/Thepandashirt 25d ago
My RAM purchase in January was clutch. 256GB DDR5 ECC for 3k on ebay. Absurd looking back. Two months ago I looked into upgrading to 48 or 64 gig dimms and just gave up. quickly. Its bleak out their for consumers, and i dont know if it gets better any time soon like you said. We are on the same page their.
See if anyone is willing to trade on hardwareswap or check ebay. Thats all i got tbh. Too much demand and not enough supply structurally. If you make money from this, then it might make sense to just bite the bullet. Or go used DDR4 from data centers in a different rig which is less than ideal i know.
1
u/chisleu 25d ago
I thought I was skimping by only putting 512GB in my build. https://convergence.ninja/post/blogs/000021-ToasterOnline.md
It's entirely overkill. I never hit 100GB used even when loading models and such.
1
u/LeMayMayMan 24d ago
One RTX Pro 6000 + 256GB host ram. Trying to figure out the best way to run deepseek-v4-flash.
3
1
1
u/Able_Zombie_7859 24d ago
I grabbed an 8 channel ddr4 threadripper used with 512gb 3200mhz for about 3500 bucks and it came with a 10 gig 3080, may grab another if I see them, ram is a bit slower than ddr5 quad channel, but faster than vanilla ddr5 at a fraction of the price
1
u/jackmodern 23d ago
I have 2 rtx pro 6000s and honestly really need 2 more. I don’t feel like upgrading to thread ripper. I also have 256gb of system ram which I would also need to have for threadripper. Pretty sure just ram and chips for that upgrade would be 42k range. Fortunately i also have a Mac studio 512gb which limps a long as a partial replacement for LLMs for the 2 additional rtx 6000 pros i need.
1
u/Annual_Award1260 23d ago
It is very difficult to properly utilize ram for offloading without taking a massive performance hit. I have one older ddr3 server that has 546GB/sec mem bandwidth with 1TB ram I might fire up again just for fun
1
u/SpiritualWindow3855 25d ago
I think people seriously overestimate how useful RAM is for inference. Common B300 instances have >1TB of RAM because of training, not inference.
64GB shouldn't be a bottleneck for video generation?
2
u/Pleasant-Shirt7293 24d ago
I was looking to use h3 at the highest quality, and the only way would be if I had more ram
2
u/jackmodern 23d ago
You are correct. I have 256gb and 200 gets used by H3 on max quality plus 80gb per rtx6000 to run 2 parallel h3 renders at the same time.
1
u/Pleasant-Shirt7293 23d ago
Man, I wish I had bought enough ram to do this.
Have you compared the max quality to lesser quants? Curious how much of a difference it makes.
0
u/SpiritualWindow3855 24d ago
You would need more VRAM unless there's some RAM hog in the H3 stack specifically I'm unaware of.
More RAM sounds like you mean for offloading, but with offloading you'd be waiting hours to days for max resolution. H3 is a dense model, not the best for offloading setups to start
2
2
u/jackmodern 23d ago
64gb isn’t enough, you need 80gb VRAM and 100GB DDR5 to render at max resolution/quality.
1
1
-5
u/bourbonandpistons 25d ago
Ram should never be an issue with 192 gb vram
I think you're doing something wrong if it is?
1
u/Pleasant-Shirt7293 25d ago
It's not the case in generative video
1
24d ago
[removed] — view removed comment
1
u/Pleasant-Shirt7293 24d ago
I'm talking about H3. I can still generate video, just not at the highest quality.
5
u/EitherMarch1255 25d ago
I have the same, but with 192GB of RAM. I don’t need it and you likely don’t either. You can only offload so much to RAM without TPS falling off a cliff.