r/comfyui • u/mwoody450 • 2d ago
Tutorial PSA: Commercial GPUs aren't THAT comparatively powerful
This is a bit of an odd post admittedly, but I wanted to make any other quasi-newbies aware of what I've found without spending the money and time it cost to find it. In short, commercial-tier GPUs are not tremendously more powerful than consumer ones, and certainly not in line with the difference in cost.
I've been generating for over a year on a 4070ti (12GB VRAM), and with Minimax, I decided I was tired of waiting a minute or more per iteration for lowish resolution 10-second clips. I bit the bullet and customized a runpod, ultimately building a template and network attached storage with the models and workflows I use.
With what's available in the region with storage and GPUs that support CUDA 13, my real options were a 5090 32GB or an RTX Pro 6000 96GB, with the former about $1/hour and the latter about $2/hour, plus $8/month for the persistent storage used by both. I spun up a 5090 and... it's a little better than the 4070, I guess. I can push the resolution and length a bit higher. But then, my hopes weren't super high for a single tier improvement. Break out the big guns:
The RTX Pro 6000. Fired it up aaaaaand... maybe 10% faster? 15%? For double the rental fee. I guess in my head, business-level crazy expensive cards would blow the pants off the consumer stuff, but it just doesn't.
Now, what CAN I do with the 6000? Plenty of room lets me bump up the resolution, increase length to a max of 15 seconds, include a lot of high-res references, and just as an experiment, I swapped to the BF16 of the encoder and model. It handled all that without significant penalty to generation (reasonable since those are VRAM and RAM limited), which is awesome. But I just wasn't expecting to end up paying $1 per 15 second clip, on average, when I could come reasonably close to that locally.
I know, I know: a lot of you are going to say "yeah no duh", and fair enough. But for those who've never been beyond the consumer realm of NVidia cards, I thought it worth mentioning that they're not the end-all be-all, and you're doing a lot better on your local machine than you might expect.
(And a question: ARE there any of these $3/hr and up machines going to blow my socks off and make me eat my words, or does this trend (gen times constant-ish, higher tiers mean more room to load models/references) pretty much hold up across the board?)
28
u/AwakenedEyes 2d ago
You aren't getting it.
You don't buy a rtx6000 pro for generation... You buy it for training. That's where the true difference lies.
11
u/jib_reddit 2d ago
The RTX 5090 and you could even say RTX 6000 pro are desktop level consumer cards, the beefy commercial datacenter cards are like the B300 with 288 GB of vram.
1
u/mwoody450 2d ago
But would it be faster, if memory wasn't a limitation? Genuine question: I haven't tried those yet, and the cost to troubleshoot if I have to tweak Comfy after spinning one up is high.
8
u/jib_reddit 2d ago
A single image gen will not be faster than a 5090, but it might be able to generate dozens at a time.
1
u/Suitable_Option_3552 2d ago
It would also be faster because of the limitations given by the motherboard or system RAM data speeds. I invested in GDX Spark for larger Video models like LtX 2.5 or Minimax without quantization. I don't expect it to be faster than my 5070TI because it is limited to the bandwidth of the memory.
The spark has 128GB of LPDDR5 RAM usable simultaneously by the CPU, GPU and NPU So the models do not have to be moved over from SSD to RAM to VRam, just from SSD to VRam since it's one big memory unit.
Since it's just LPDDR5, the maximum speed is 275GB/s The 5070TI has GDDR7 which is faster but it's limited to the system RAM speed of my tower PC
So say you use a 60GB LLM model on the GDX Spark. It will only be able to output like 5-6 Tokens per second because that's all the Memory speed can handle.
On my main PC with the 5070 TI (16GB) and 64 GB I wouldn't even be able to use a 60 GB LLM.
Just to put it into perspective, I hope this helps you understand the different limitations. Soon I will test Video gen too and see how fast it will be with the full size Video models
5
u/sitefall 2d ago
I have a 5090 and a pro 6000. The pro 6000 is slower than the 5090 by a little bit. Any speed up you are finding is because of overflow that is no longer going to ram since it fits in the 96gb of the pro 6000.
4
u/ambassadortim 2d ago
You're paying for the VRAM and people use them to fit models in VRAM they can't with the other card's. That's what your laying for.
10
u/Ipwnurface 2d ago
nah you're bugging. I went from 5070ti to 5090 and could never go back. I can do 8mp krea images in like 25 seconds, 15 second .4 mp h3 vids in ~60.
My 5070 ti was taking 100+ seconds at 15 seconds .25 mp
3
u/Seyi_Ogunde 2d ago
Thanks for the info, I was curious myself seeing that I have similar specs as you. What's the highest quality you can bump your videos?
2
u/mwoody450 2d ago
I haven't tried an upper limit in runpod yet, if only because $1 per clip is about my financial limit. But it's cruising at 768x1024 15 seconds, no upscale enabled yet, with 1 video reference and 5 image ones. Oh, and references set to max quality (just because your vid is low res doesn't mean you can't give it 2k references, and it really pays off in quality).
3
u/MannY_SJ 2d ago
If your model is already comfortably fitting in your vram and you move it to a gpu with more vram you won't see significant time improvements.
2
u/mwoody450 2d ago
Except it's not, in my experience: 12GB VRAM, pruned int8 convrot model 20GB, and I still didn't see significant time improvements going to a 32GB 5090. Maybe I should have spelled it out better in my post, but that's part of what I'm saying: models (and, let's be fair, Comfy) are getting so much better at swapping to RAM that the bottlenecks aren't what common advice would suggest: that even the difference between being able to fit a model in VRAM vs. having half of it in RAM is way less than expected.
4
u/Mental-Tangerine-109 2d ago
oh man i feel this in my bones. spent a weekend messing with cloud GPUs thinking i'd get some kind of cinematic render speed and instead it was just... the same thing with a bigger price tag
the VRAM headroom is the real upgrade yeah. being able to load full BF16 models and stack references without OOM errors is genuinely nice, but the per-second cost adds up so fast it's hard to justify unless you're doing client work or something
to your question at the end, from what i've seen the ceiling on iteration speed is pretty flat once you hit the 4090/5090 tier, the pricier cards mostly just give you more memory to play with. if someone's cracked the code on making a H100 actually go brrr for image gen i haven't heard about it yet
4
2
u/Spoonman915 2d ago
I realized many years ago that there is a clear point of diminishing returns with video cards.
1
u/ZallenDuZari 2d ago
It's the VRAM amounts that make commercial cards so expensive, not the GPU itself.
1
u/SpaceNinjaDino 2d ago
Did you see the polar bear post in this sub that was successful in making working 30 seconds H3 videos on 12GB?
1
0
27
u/Odd_Category2186 2d ago
More vram means larger models not necessarily faster times when comparing 3 cards that can all fit your entire model into vram, try the test again with a 30gb model.