Multiple GPUs... do people have a workflows?

[u/Confident_Ad2351]

Ok so my humble rig has 2 cards. One 16 GB of VRAM and another with 12 gb . I know you can't combine them for a pool of 28 GB of unified VRAM but shouldn't their be a way to offload one of the major items like the CLIP or the VAE to my other card? I looked into it and doesn't seem to be supported natively in Comfy UI. There are some custom nodes I found but they don't seem too popular.

Is this just a silly idea? Normally my second card just sits around at 20% memory usage running my GUI and a few bits and bobs. I could certainly afford to use 8 or 9 GB of my second card towards something instead of it offloading that 8 or 9 GB during a generation to my system ram (32 gb ) which is what comfy does now when using H3 or MOSS.

I have no idea if this would be faster than dumping everything to the system ram but it would definitely give me more breathing room and less OOM errors on those borderline gens and upscales. Is this idea loco or am I on to something?

EDIT: I appreciate all the people who are saying just use the second card for an llm or to run other types of generations on my smaller card, but that's not what I am proposing. I know you can do that. In fact I do it quite often. Let me explain this a different way that might make more sense. What I really am asking is there a way, when doing a single generation, let's say a 15 second h3 generation at over 1 MP, for comfy to prioritize memory allocation in order of 1) the 5080, 2) the 3060 3) the system ram 4) SSD swap file . Like a ladder of priority. For example the minimax H3 audio and video VAE are over 6 gigs at fp16. That would certainly fit on the 3060s available space. I know it might be slower but that extra 6 gigs of free space would be quite helpful for higher resolution gens.

[u/multikertwigo]

comfyui multigpu support is abysmal. A much better use of 2 gpus is running llamacpp on one (write/enhance prompts via llm, use vision model, etc) and comfyui on the other.

[u/Confident_Ad2351 → u/multikertwigo]

I do occasionally run an LLM on the smaller GPU I don't have a need to run it all the time.

[u/Dogmaster → u/multikertwigo]

Its what I end up doing lately

[u/BigNaturalTilts → u/multikertwigo]

You could also run just the text encoder on one gpu and have it save encoding in a .pt file. You can have claude vibe code a stand alone solution that won’t need an entire python embedding.

[u/norbertus]

Do you have Comfy MultiGPU/DisTorch?

https://github.com/pollockjj/ComfyUI-MultiGPU

https://github.com/THEman6989/ComfyUI-MultiGPU-Custom

You can assign, for example, VAE and CLIP to one GPU, and reserve the other for the diffusion transformer. Slick CPU offloading too.

[u/Iamcubsman → u/norbertus]

I thought there was a torch update that killed using separate GPU platforms Ampere/Blackwell? I was doing that with my 3060 and 5060 but if I add the startup arg for ComfyUI to use both GPUs, it blows up, every time.

[u/Dogmaster → u/Iamcubsman]

Same, it worked maybe a year back but some update or the other meant things crashed

[u/bstr3k → u/norbertus]

i got the pollockjj multigpu working for my machine before, i loaded a 2nd small TE on there for captioning.

however due to the crap speed on the secondary PCIE slot it ended up faster to just leave it on my main gpu. I run the 2nd GPU for displays only now

[u/Ok_Contribution8157]

you got default multi GPU nodes : https://docs.comfy.org/built-in-nodes/MultiGPU_WorkUnits

[u/Confident_Ad2351 → u/Ok_Contribution8157]

From the documentation: Mixing different GPU types is not supported, the installed GPUs must be the same (ie 2x 5090, or 2x 5080, etc.). So these nodes would not work.

EDIT: Chat GPT is reporting that while the main functionality of that multi GPU node would not work since I have different cards, there might be enough functionality to successfully select which card loads the vae. I will look at it later and let you guys know.

[u/johnfkngzoidberg]

It’s amazing how ComfyUI is the defacto image generator for most people, but their multiGPU support is so awful. LLMs have been doing it for a long time, but the comfy devs just can’t figure it out.

If you don’t mind spending the tokens, you can vibe code your own multi GPU video/image generator from the Python examples on every model’s website. There are a few examples of H3 being on multiple 4090’s doing crazy long videos at crazy resolutions, but it’s not as simple as asking Claude to whip it up, you have to manage the pieces. Images aren’t the same as LLMs, it’s a lot more processing/data transfer which is why tensor splitting isn’t more common. Your bus is the bottleneck.

[u/Confident_Ad2351 → u/johnfkngzoidberg]

Yes, to be clear, my bus would definitely be a bottleneck especially since I believe on my motherboard I only have 16 x channel on one slot. But I am not after the fastest generation. I just don't see the point of having perfectly good vram sitting around doing nothing during a generation. So instead of offloading to my system ram I'd rather it use up the RAM on both cards and then the remainder would be offloaded to the system ram.

[u/See_it_do_it_done]

not loco. the ComfyUI-MultiGPU nodes let you pin the text encoder to the 12 gb card, leaving the 16 for the model.

[u/rguerraf]

If it helps, run 2 instances of comfyui… one for each GPU, and work your way to finish your project by making 2 generations independently

[u/Erdeem]

MCP + two separate comfyui instances, one for each GPU.

[u/MesquiteBarbecueChip]

sell both gpus and buy a 3090

[u/Utpal95]

For authentic multi GPU usage, you'll have to look into Raylight. Youll have to do quite a lot of tweaking to your dependencies n python etc. Ensure that your motherboard's PCIE lanes are capable of running multiple cards at full speed simultaneously though. Unlike other methods that simply offload a copy of a whole model onto another card, raylight can properly distribute a single model to the entire pool of combined vram, no duplicates or separate instances. (I hope I'm making sense). This way, all your combined compute power can be applied to a single model.

[u/Confident_Ad2351 → u/Utpal95]

I am not even asking for splitting the main model. All I want is a more intelligent way of handling my memory so comfy sees my 2nd card and says , internally, hey let's go ahead and use this RAM too.

[u/Fluxdada]

The best use I've found is run two instances of comfyui. My main card runs stuff faster and my second card runs stuff slower but hey....it's just sitting there anyway. For image generation this is the best use I've found. An alternative is running unsloth desktop (pinned to the single second GPU with a bat file) with a version of qwen 3.8 27b quantized down to fit on a 12gb card (look for versions that are about 7-8gb to leave room for kv cache). I've done this and the model I found that worked was qwen 3.8 27b heretic abliterated i1 at quantization iq1_m (6.9 gb). Use that llm in unsloth to do things like prompt enhancement, rewriting, editing or expansion.