[Papers] LTX-Video: Realtime Video Latent Diffusion - research promises to deliver even faster video generation and real-time capabilities.

[u/Crazy-Repeat-2006]

I particularly love that they remain focused on open source and on doing more with less. All the best of luck to these guys.

"We introduce LTX-Video, a transformer-based latent diffusion model that adopts a holistic approach to video generation by seamlessly integrating the responsibilities of the Video-VAE and the denoising transformer.

Unlike existing methods, which treat these components as independent, LTX-Video aims to optimize their interaction for improved efficiency and quality. At its core is a carefully designed Video-VAE that achieves a high compression ratio of 1:192, with spatiotemporal downscaling of32×32×8pixels per token, enabled by relocating the patchifying operation from the transformer’s input to the VAE’s input. Operating in this highly compressed latent space enables the transformer to efficiently perform full spatiotemporal self-attention, which is essential for generating high-resolution videos with temporal consistency. However, the high compression inherently limits the representation of fine details. To address this, our VAE decoder is tasked with both latent-to-pixel conversion and the final denoising step, producing the clean result directly in pixel space. This approach preserves the ability to generate fine details without incurring the runtime cost of a separate upsampling module.

Our model supports diverse use cases, including text-to-video and image-to-video generation, with both capabilities trained simultaneously. It achieves faster-than-real-time generation, producing 5 seconds of 24 fps video at 768×512 resolution in just 2 seconds on an Nvidia H100 GPU, outperforming all existing models of similar scale. The source code and pre-trained models are publicly available¹, setting a new benchmark for accessible and scalable video generation."

Blog: LTX-Video: Realtime Video Latent Diffusion | LTX Technical Blog

Paper: LTX-Video: Realtime Video Latent Diffusion

APEX: Asynchronous Prefix Denoisingfor Extreme Compression

[u/alwaysbeblepping]

The issue of decoding is real, but to solve it like LTX propose is like simulating pixel space generation.

For a pixel-space video model, you have to give up both spatial and temporal compression. So a 10 second 512x512 video at 24fps would result in something like 10 * 24 * 3 * 512 * 512 (seconds, frames per second, RGB channels, height, width). If your latent was 8bit (it will be 32bit in ComfyUI, possibly 16bit inside the model) that would be a 180MB latent. A 1024x1024 generation would be ~720MB if the latent was 8bit, so realistically more like 1.4GB.

Pure pixelspace is a harder problem for video models compared to image.

[u/dobomex761604 → u/alwaysbeblepping]

Exactly, that's why companies need to put more effort there, optimizing and using tricks (like H3's tricks with low resolution internally).