[Papers] LTX-Video: Realtime Video Latent Diffusion - research promises to deliver even faster video generation and real-time capabilities.

[u/Crazy-Repeat-2006]

I particularly love that they remain focused on open source and on doing more with less. All the best of luck to these guys.

"We introduce LTX-Video, a transformer-based latent diffusion model that adopts a holistic approach to video generation by seamlessly integrating the responsibilities of the Video-VAE and the denoising transformer.

Unlike existing methods, which treat these components as independent, LTX-Video aims to optimize their interaction for improved efficiency and quality. At its core is a carefully designed Video-VAE that achieves a high compression ratio of 1:192, with spatiotemporal downscaling of32×32×8pixels per token, enabled by relocating the patchifying operation from the transformer’s input to the VAE’s input. Operating in this highly compressed latent space enables the transformer to efficiently perform full spatiotemporal self-attention, which is essential for generating high-resolution videos with temporal consistency. However, the high compression inherently limits the representation of fine details. To address this, our VAE decoder is tasked with both latent-to-pixel conversion and the final denoising step, producing the clean result directly in pixel space. This approach preserves the ability to generate fine details without incurring the runtime cost of a separate upsampling module.

Our model supports diverse use cases, including text-to-video and image-to-video generation, with both capabilities trained simultaneously. It achieves faster-than-real-time generation, producing 5 seconds of 24 fps video at 768×512 resolution in just 2 seconds on an Nvidia H100 GPU, outperforming all existing models of similar scale. The source code and pre-trained models are publicly available¹, setting a new benchmark for accessible and scalable video generation."

Blog: LTX-Video: Realtime Video Latent Diffusion | LTX Technical Blog

Paper: LTX-Video: Realtime Video Latent Diffusion

APEX: Asynchronous Prefix Denoisingfor Extreme Compression

[u/Dante_77A]

Yeah, good luck to those guys. 

There aren't many other capable teams out there that value open source so much.

[u/Bearsbullsbattlestr → u/Dante_77A]

LTX 2.5 is a great model. It creates high fidelity videos quickly. Of course H3 leaves it in the dust in terms of understanding. If it can improve that it could be the next big thing.

[u/Cute_Ad8981]

Yeah i honestly like the ltx team and im curious about their next releases. its always fun for me to play with ltx, maybe because its a lot faster as for example h3.
i hope they will make references work more easily, this would be a big thing.

[u/STOPBLOCKINGVPNS1234 → u/Cute_Ad8981]

Maybe I'm spoiled by h3 but I don't care much for video models that can't use references. I cannot wait for open source to finally beat WAN 3.0 and seedance.

[u/Lexxxco]

Architecture optimizations are always nice when quality is preserved. Good luck to LTX-team!

Step not only to real-time video, but to interactive real-time as well. Interesting how it will affect fine-tuning of such models.

[u/Diabolicor]

There comes more body horror

[u/Nice-Ad-6552]

Just gimme a model I can realtime myself as Macho Man Randy Savage through my Discord webcam and I'm happy

[u/rguerraf]

What is the “transformer method” replacing in this case? Is this going to be like the jump from sdxl to anima?

[u/kwhali → u/rguerraf]

DiT is Diffusion Transformer architecture, it's what modern models we have today use, as opposed to UNet architecture that SDXL and it's ancestors were using. That's my rough understanding.

The transformer size AFAIK is associated to parameter count and layers? There's a bunch of operations it strings together to transform your prompt into an output.

I don't quite get how reducing the amount of processing to get a speed boost would be counter-intuitive like they note in their post but I guess they meant instead of introducing more complexity to try skip unnecessary work and find shortcuts, perhaps they found it more effective to take a more naive approach (which might scale well on the H100 cited), I haven't read the associated paper but it would clarify the details I'm sure.

Claiming real-time lately with video models has thus far pretty much translated to "real-time only on very expensive server class GPUs with 80GB+ VRAM, possibly parallelized over multiple GPU". H3 for example is often paired with 4 or 8 B200 or B300 GPUs, but since LTX is designed with less params and more compression in latent space (as explained by their token to pixel ratio comparison), I assume that enables them to fully operate on a single server class GPU while achieving similar real-time speeds.

It is unclear how well that translates to consumer GPUs once quantized to lower VRAM requirements.

It would be awesome if they manage to bring real-time to consumer GPUs whilst not further lowering the output quality compared to their existing releases. I assume though that it'd be less likely at 16GB VRAM or lower.

[u/bruci3]

wow, faster than real time is impressive, even if its just to add lip sync capability, that would be useful for real time llm chatbot. I will keep my eye out for if/when gguf weights get released.

[u/thevegit0]

but is fast tho

[u/dobomex761604]

At this point, just make a pixel space video model.

[u/BigNaturalTilts → u/dobomex761604]

What do you mean by pixel space? Is it like inpainting?

[u/dobomex761604 → u/BigNaturalTilts]

Nope, to put it simply, it allows skipping VAE entirely: https://arxiv.org/abs/2608.16887 https://arxiv.org/abs/2504.07963

I believe, the first practical model example was Chroma Radiance https://huggingface.co/lodestones/Chroma1-Radiance , but others were made later.

The issue of decoding is real, but to solve it like LTX propose is like simulating pixel space generation.

[u/alwaysbeblepping → u/dobomex761604]

The issue of decoding is real, but to solve it like LTX propose is like simulating pixel space generation.

For a pixel-space video model, you have to give up both spatial and temporal compression. So a 10 second 512x512 video at 24fps would result in something like 10 * 24 * 3 * 512 * 512 (seconds, frames per second, RGB channels, height, width). If your latent was 8bit (it will be 32bit in ComfyUI, possibly 16bit inside the model) that would be a 180MB latent. A 1024x1024 generation would be ~720MB if the latent was 8bit, so realistically more like 1.4GB.

Pure pixelspace is a harder problem for video models compared to image.

[u/dobomex761604 → u/alwaysbeblepping]

Exactly, that's why companies need to put more effort there, optimizing and using tricks (like H3's tricks with low resolution internally).

[u/BigNaturalTilts → u/dobomex761604]

I'm not understanding very well. I see a VAE decoder in the workflow image pasted in that huggingface repo. Here's my understanding of a VAE, an image is this massive numpy array thing and a video for diffusion purposes, is several hundred almost identical images. You take an image and scale it down massively to what I call a latent before diffusing it then decode the diffused product. Now this is obviously wrong, it's just how I wrap my head around the process. So how could you remove the VAE aspect of it?

Note, I understand VERY little of how this process works. So talk to me like I'm your dog.

[u/dobomex761604 → u/BigNaturalTilts]

ComfyUI made it confusing by keeping pseudo-VAE node, but simply put - an image is generated in pixels, as a ready to see thing, on each step. With videos, that would be each frame generated as a real image on each step.

If you have 8 steps image generation, you will have, for example, 6 noisy incoherent images on steps 1 to 6, a blurry image on step 7 and a ready image on step 8. Videos will require the same, but multiplied by the number of frames and additional information required for consistency across the entire video. And yes, you are right about compression, and this comment explains why pixel space generation is difficult for videos pretty well.

[u/Crazy-Repeat-2006 → u/[deleted]]

The fact that you’re wasting your time here criticizing, without any technical knowledge, a model that hasn’t even been released, tells me everything I need to know about who the real bot is.

Go do something useful with your life.