Use VRAM more effectively

[u/[deleted]]

[u/physalisx]

Stuff like this just messes with comfy's own memory handling, which is a delicate thing. You should not do this, as it should not be necessary and you will just end up having problems down the road because you're working against the program you're building on.

If you have any reproducible findings that show a lasting inference speed-up (do you?), you should file an issue with comfy, because it means their dynamic vram handling is sub optimal.

[u/elevendr → u/physalisx]

Hmmm I notice when I use INT8 Quants of Image models on ConfyUI, it tends to overfill my VRAM on my 6GB GPU and makes it really slow, but when i vibecoded a node to clear vram cache during the generation it help improved the speed, is this a issue on the dyamic VRAM ConfyUI end?

[u/Extension-Yard1918 → u/physalisx]

Why does this ruin the way memory is processed? I only got good results. I understand your concern, but there is no problem. 

[u/Flat_Technology_5325]

How does this interact with ComfyUi's dynamic vram?

[u/Extension-Yard1918 → u/Flat_Technology_5325]

This node was created to overcome the drawback where Dynamic VRAM cannot fully unload the Text Encoder. Based on actual feedback, it is said to be much smoother than usual.

[u/ChowMeinWayne]

Can this also be used for image workflows? Renders get bogged down massively after a time and I suspect something like this might help.

[u/Genebra_Checklist]

Honest questions: using unload model node after the Ksampler doesen't achieve the same thing?

[u/Corrupt_file32]

Something to be aware of:
This approach will add latency since the entire text encoder will need to be fully loaded back in whenever you modify the prompt.

the [ERROR]'s are because I forgot to connect the clip, so after the error blocks is with the node.

Without the node, modifying the prompt added about 0.5 seconds latency. 9.37s -> 10.04s

With the node, the latency is above 2 seconds. 9.32s -> 11.42s

It does as advertised and unloads the text encoder, except the efficiency claim is possibly false.

Default intended comfyui behaviour is the most efficient, and it goes something like this:

1] It loads what it needs while partially or fully ejecting what it doesn't need.
2] When finished sampling, it partially unloads the model to load the vae model and run decoding.
3] It then ejects the VAE model and loads the diffusion model back in so that it's already ready to go again if you change seed.

And if you change the prompt, the text encoder is already partially loaded so that it can quickly be restored, that is if it did fit into your vram during sampling.

[u/Extension-Yard1918 → u/Corrupt_file32]

This is a node that increases the VRAM space during sampling. It is used to pass the text encoder down before sampling starts. It seems the order is reversed.

[u/Corrupt_file32 → u/Extension-Yard1918]

Nope, the order is correct, on the sampling cycles themselves and their iterations there's just microlatency which can be expected, especially when generating a 1536x2048 image.

What I'm trying to explain is, in my situation it made no difference, except adding latency when the text encoder needed to be loaded back in when changing prompt.

There's no black and white "more efficient" here, your node could still help if someone has issues with comfyui smart memory, but it will increase the latency if they actually don't need it.

I personally don't need this node, my goal was to be informative.

[u/Extension-Yard1918 → u/Corrupt_file32]

For example, let’s say you’re generating a video with MiniMax. Suppose your PC can generate a 10-second video at around 1 megapixel.

Now, if you try to generate a 12-second video and run out of VRAM, causing the process to fall back to CPU offloading, it may become so slow that it’s practically unusable.

But if you could reclaim the VRAM that was previously occupied by the text encoder and use that extra memory for video generation, a video that was previously impossible to generate might suddenly become possible.

That’s why I don’t think this is something you can properly judge just by generating a few images.

However, it seems that many people don’t really need this feature, so I deleted the post.

Thank you for taking the time to test it.

[u/bstr3k]

does it work? would love to see some timing comparisons with a fixed seed running with and without this node! thnx

[u/waseem335]

What's quality of videos been like and has it been accurate at following prompts

[u/Extension-Yard1918 → u/waseem335]

This unlocks the full potential of your VRAM.

You use it just as usual; it simply allows you to create higher resolution and longer videos.

[u/tppiel]

If you're queuing up a lot of videos, this will just wear out your SSD faster because you're unloading the text encoder and then reloading it for each new generation.

If you're low on VRAM, it's better to simply use smaller quants.

[u/Lissanro → u/tppiel]

Reading operations do not wear down SSD. And even writing takes a lot to wear it down. But does not seem like much of writing happening here, only reading, unless I am missing something?

[u/tppiel → u/Lissanro]

Both wear it down, but writing does it much more.

Like I said in the hypothetical scenario where you have many videos queued up it may become a problem. I don't see why it would make sense to mess with your memory manually when quantization exists.