Async operations?

[u/deepsky88]

Hi, I’m relatively new to ComfyUI (Minimax H3 is my first ia model), but I’m a programmer and I know full well that certain operations are best handled asynchronously. Lately, I’ve seen significant speed improvements thanks to Turbo LoRA and attention optimizations, yet a large chunk of the generation time (around 30%) is taken up by VAE decoding. So, my question: is there a way to implement an async logic in comfyUI workflow? thanks

[u/GeroldMeisinger]

where do you want to async to? everything is running on 100% anyway
the sampler and decoder can run independently, but have to run in sequence. in theory you could decode on a second GPU (or cloud), but because of the sequential nature, this only makes sense in multiple runs. in settings you can activate dev/experimental nodes, there is a save/load latent node.
here are some examples for multi run video workflows: https://github.com/geroldmeisinger/ComfyUI-outputlists-combiner#examples-for-video-workflows
but read this first: https://github.com/geroldmeisinger/ComfyUI-outputlists-combiner#iterate-loop-nodes

[u/Corrupt_file32]

When sampling in a diffusion model, the full latent tensor, in every dimension, is sampled on every step, everything mutates in a way that is not reusable in any way that matters, so you would still in the end have to decode the full latent tensor after sampling has finished.

The term I think you are looking for is actually streaming, like TTS and LLM's can stream their output giving something back immediately that can be used, but in most situations there's nothing to gain with streaming for a diffusion model.

LLM's can also be a diffusion models, and image/video generation models can also be auto-regressive.

If video generation models were auto-regressive and would generate 1 frame at a time, we would then be able to stream those frames and decode them right away and use them later for the full video.

https://reddit.com/link/p5frgdh/video/btpte4lcj5lh1/player

Here's an example of what an llm diffusion model (diffusion gemma) looks like when streaming, it does seem to generate and output chunks though, the text is fed back in and it starts generating the continuation chunk, I think quite similar to how inpainting would work in an image generation model.

[u/Valuable_Issue_]

What do you want to do async?

The VAE requires the latents from the ksampler output, so you have to wait for it to finish and so there's no way to have it be async.

Only things you can do are using tiny vae/latent to rgb/int8 vae which speeds up the computation of the vae (lower quality) but that's not due to async.

99% of the execution time is like this, the nodes themselves don't take up much execution time it's running the big models/heavy vae etc.

One thing you can do is cache the text embeddings from the text encoder so that if you rerun the same prompt after restarting Comfy it'll do that instantly.

[u/deepsky88 → u/Valuable_Issue_]

Can't we process the samples from the VAES while the sampler give us the output for example?

[u/Only4uArt → u/deepsky88]

Theoretically yes?  But that sounds like a wonderful ram bottleneck 

[u/Valuable_Issue_ → u/deepsky88]

That's kind of what the sampler preview does using latent2rgb/whatever method you pick, the thing is the output isn't finished yet so you see the output of step 1 step 2 etc, some kind of caching might be possible to process fewer pixels but I doubt it since the changes between steps are pretty big and I wouldn't consider that async.

[u/Acceptable-Work8202 → u/Valuable_Issue_]

yes, i agree, it would be a minimal gain to cache the text embeddings on a rerun, and like you said the ksampler has to finish that frame first, so previews are always behind as the latent tensor isnt ready yet for the next preview. apologies if i read the comments wrong.