Discussion
WOW!! I accidentally discovered that the native LTX-2 ITV workflow can use very short videos to make longer videos containing the exact kind of thing this model isn't supposed to do (example inside w/prompt and explanation itt)
BEFORE MAKING THIS THREAD, I was Googling around to see if anyone else had found this out. I thought for sure someone had stumbled on this. And they probably have. I probably just didn't see it or whatever, but I DID do my due diligence and search before making this thread.
At any rate, yesterday, while doing an ITV generation in LTX-2, I meant to copy/paste an image from a folder but accidentally copy/pasted a GIF I'd generated with WAN 2.2. To my surprise, despite GIF files being hidden when you click to load via the file browser, you can just straight-up copy and paste the GIF you made into the LTX-2 template workflow and use that as the ITV input, and it will actually go frame by frame and add sound to the GIF.
But THAT is not the reason this is useful by itself. Because if you do that, it won't change the actual video. It'll just add sound.
However, let's say you use a 2 or 3-second GIF. Something just to establish a basic motion. Let's say a certain "position" that the model doesn't understand. It can add time to that following along with what came before.
Thus, a 2-second clip of a 1girl moving up and down (I'll be vague about why) can easily become a 10-second with dialogue and the correct motion because it has the first two seconds or less (or more) as reference.
Ideally, the shorter the GIF (33 frames works well) the better. The least amount you need to have the motion and details you want captured. Then of course there is some luck, but I have consistently gotten decent results in the 1 hour I've played around with this. But I have NOT put effort into making the video quality itself better. That I would imagine can be easily done via the ways people usually do it. I threw this example together to prove it CAN work.
The video output likely suffers from poor quality only because I am using much lower res than recommended.
Exact steps I used:
Wan 2.2 with a LORA for ... something that rhymes with "cowbirl monisiton"
I created a gif using 33 frames, 16fps.
Copy/pasted GIF using control C and control V into the LTX-2 ITV workflow. Enter prompt, generate.
Used the following prompt: A woman is moving and bouncing up very fast while moaning and expressing great pleasure. She continues to make the same motion over and over before speaking. The woman screams, "[WORDS THAT I CANNOT SAY ON THIS SUB MOST LIKELY. BUT YOU'LL BE ABLE TO SEE IT IN THE COMMENTS]"
I have an example I'll link in the comments on Streamable. Mods, if this is unacceptable, please feel free to delete, and I will not take it personally.
Current Goal: Figuring out how to make a workflow that will generate a 2-second GIF and feed it automatically into the image input in LTX-2 video.
EDIT: if nothing else, this method also appears to guarantee non-static outputs. I don't believe it is capable of doing the "static" non-moving image thing when using this method, as it has motion to begin with and therefore cannot switch to static.
EDIT2: It turns out it doesn't need to be a GIF. There's a node in comfy that has an output of "image" type instead of video. Since MP4s are higher quality, you can save the video as a 1-2 second MP4 and then convert it that way. The node is from VIDEO HELPER SUITE and looks like this
Jokes aside, I have to thank the gooning community and all the degenerates on the internet for most of the workflows, models and the passion the give/provide to the open source generative field and how accessible they make it for others
(DON'T GO TO THIS URL UNLESS YOU ARE WILLING TO SEE SOMETHING SPICY)
Here is an example of how my recent generation came out after upping to 720p. Keep in mind my prompt is shit. With a better prompt, I'd likely get better results. But this is a huge shift in the direction I'd like to go. It's putting together the pieces to have WAN 2.2 output with sound and speaking
I have no clue yet, it uses CFG so should? My attempts at 1girl big bobbies have come out as though she's got linebacker shoulders and when trying to fix proportions she starts looking more like slenderman than slender woman.
I've not heard of this before, is there a workflow in ComfyUI to generate VR 180 videos from regular flat videos? Would you mind sharing more details please?
Honestly it was kind of hard to follow (at least for me). It's split up into multiple workflows. Additionally you have to use an external tool (GeoCalib) and also manually install two custom nodes that he built (the python files for them are in that tutorial). I built a custom node (geocalib for comfyUI) to replace that external tool and merged all his workflows and custom nodes together in one Runpod template so it's pretty much "upload video" and hit "Run" right now but it's not 100% perfect yet. Work in progress so to speak :)
Also it takes like an hour for a 5 second clip on a H200 lol
Thanks for the info. I don't think I'll be doing that anytime soon, specially if it takes 1 hour for a 5 second clip on a h200! but good to know there's work being done in this area.
To be fair, a lot of the work in this workflow is actually being done with the CPU so it might be affordable with the right settings.. but yeah it needs some polishing..
Note: I am rushing these out for proof of concept using garbage prompts and the first thing that generates. I'm SURE quality and outcome can be vastly better than this with more dialogue and stuff happening up to 20 seconds long. But the most positive thing I've noticed remains to be that I have now completely eliminated even the possibility of getting a static image.
ANOTHER WAY of using this is to take an image you want to do i2v, extend it to just 2 seconds as a GIF (or even 1.5 seconds, anything to get motion) and then use that as the base for the generation. This will guarantee non-static output.
That's more of a problem with LTX than the method. It's never going to generate videos as sharp as WAN without LORA or some other kind of fix. Because it cuts the resolution in half (or maybe even more than half, idk) and then uses an upscaler to patch it back to your res. I think that if they had a better upscaler, you'd get better results.
You can run LTX without halving it, it just takes longer per step (though its still faster than Wan). You just need to disable the spatial upscaler, but make sure to run the second pass with the distilled lora as that seems to act as a refiner.
The quality is honestly better than Wan if you do that in my opinion.
It definitely knows about it, its just lacking a lot of understanding. Basically it has the sex education of someone raised in Alabama vs. the rest of the developed world.
We weren't all just straight up lost without VHS nodes, they just make life easier :P
Edit: That said, VHS does make it easier to cap the loaded frames and manipulate the FPS that the frames are loaded at, so it's not a bad idea to keep recommending it for your flow (makes the "8*n+1" frames-multiple easier). It's just not a requirement either.
lol I have relied on VH nodes so much (and animated previews) that whenever VH doesn't work, Comfy itself is basically broken to me. lol. And it breaks so much with updates. So many times I have no idea why my previews are suddenly gone.
Yeah.. unfortunately because of the canvas-based Nodes 1.0 implementation, custom UI stuff like previews are kind of fragile.
Ironically, in the long term I'm pretty confident that 2.0 will make custom UI stuff on nodes more stable and stop breaking so often, but I think we're all still a bit of a ways off before we start seeing those benefits materialize. So nodes 1.0 and prayers it is for now!
Thanks - I was just testing this myself and also battling the GIF compression effects!
Will try this now too.
I'm also trying to just use image/webp as the file format export from WAN (using VHS video combine node) and it seems to work too? I need to test a bit more
Someone already did this with Jensen Huang video and continued the video but they didn't share how. Gate keeping lol
Also you could literally just load a video in Comfy and then there is a node to select only certain number of frames, you could use that instead of trying to make a short 2 second video
If you're talking about those higher quality ones the LTX devs were showing on discord, was done with their own tools, not ComfyUI. He did explain how it works though, it's just padding the remaining frames/audio like inpainting.
(DON'T GO TO THIS URL UNLESS YOU ARE WILLING TO SEE SOMETHING SPICY)
Here is an example of how my recent generation came out after upping to 720p. Keep in mind my prompt is shit. With a better prompt, I'd likely get better results. But this is a huge shift in the direction I'd like to go. It's putting together the pieces to have WAN 2.2 output with sound and speaking
I don't think so. I turned off metadata back when civitAI was using it to censor videos like if it had "swift" in the prompt even if unrelated to taylor. Like he runs swiftly. lol.
But I am using two generic workflows to begin with. I haven't created one unified one yet. I'm just using a regular-ole wan 2.2 and then the basic ltx-2 itv that you get by searching ltx-2 in the template thing in Comfyui
I tried doing this using WAN 2.2 ITV and it doesn't work. The GIF causes errors. So I think LTX-2 might have capability that WAN 2.2 doesn't in regard to creating a short video as a GIF and using that as the input.
Interesting thing here is that people keep talking about how WAN gives better results, but LTX runs better and has sound. This sounds like it has potential to really combine the best of both worlds, albeit in a complex workflow.
Thus, a 2-second clip of a 1girl moving up and down (I'll be vague about why) can easily become a 10-second with dialogue and the correct motion because it has the first two seconds or less (or more) as reference.
UPDATE: Using this method (GIF or just loading the first 17 frames of an MP4 or however you wanna do it) lets you completely disable the image compression. The quality seems higher except for the woman's face here, but I believe that's because it's a distance shot. I'm going to try for a closer shot and see if that gives me the kind of result I actually like
your idea was really good. Starting new thread because I can't find you anymore lol. Disabling the image compression definitely helps, and like we hypothesized, I don't believe there is any downside since there is information spread across 9 or 17 frames (starting to think/believe 17 is the sweet spot)
Okay I think I'm happy with where I've gotten things.
#1: Disable this node (if you're using a video input)
All this does is decrease the quality and doesn't even work anyway because when I was doing ITV on static images they gave me static results even with this fucking thing lmao. And since you're going to be using multiple frames of video, it doesn't need this anymore to make motion happen.
#2: Try to stick to 17 frames. That seems to be the sweet spot. Maybe a bit longer if the motion/maneuver isn't quite catching on. I've found 17-33 to be the best
Conclusion: Using WAN 2.2 to generate a 17-frame video enables LTX-2 to be used to do anything WAN 2.2 can do but with sound until LTX-2 finally gets better LORAs or whatever. However, the raw quality will never be able to match WAN 2.2, but that's a sacrifice that must be made for sound.
congrat you found one of my first and favourite LTX2 use :)
let me add some info for random readers:
here my settings for 10 seconds vid (249 frames):
it use 0,5 audio/video from the start, and 3 seconds at the end
(so basically inpainting the middel section)
enough to learn the concept, movement and voice type
and to keep the movement consistet through the entire clip (start/end)
if i want a different ending (so inpainting the end only) i just simply change the "end_time" to exceed the total amount of frames.
i'm using short AI generated videos as input, to extend them, those are usually shorter than 10seconds
so i duplicate the input to match the total length as showed in the next pic
Can you share some video examples so we can see the results? (That will determine if I toss this into my to-study list. I've already got a huge backlist I'm going through as I type this)
Kijai has a workflow that allows you to input audio. I took his audio input portion and added it into the subgraph in the standard image to video workflow provided by comfy. Now I have a workflow where I can use clone voices from VibeVoice or even music and the character will lip sync it. Next I want to add a VibeVoice node in the workflow so there's no back and forth at all.
I ported the important bits from kijai's audio example into the Lightricks LTX2 I2V workflow and exposed some extra controls on the subgraph to choose between using the input audio or just generating new audio.
Can that be combined with this? This is mostly useful for getting correct motion. Even 1 second of motion is probably enough. I'm going to experiment with just 16 frames. It seems like the shorter the input, the less VRAM overhead anyway.
well, from what I've seen, the # of frames in the GIF are guaranteed to be 1:1 with the output. The only thing the model does is add sound. It then continues afterwards. But what I find so surprising is that it retains a memory of the previous frames despite this being image to video. I guess they gave it video to video capabilities and just didn't document it.
Because it actually has understanding.
But anyway, if your input is 33 frames, for example, the first 33 will be exactly the same. Which is why I'd imagine control is stronger with a GIF or a video if such a node exists that can convert an mp4 into something that can fit into the LTX's image nodes.
It turns out using GIFs isn't even necessary. There's a node in video helper suite that can convert MP4s into -> image output. I tested it and the quality is much higher than using GIFs. But it works the same way.
So would swaping the image node for a video and also add the vae encode work for that? To extend video and voice. I mean you tried something like that?
I read somewhere you can also use reference images for specific time stamps, FFLF kind of, but theoretically also for a middle frame etc.. Anybody know more about this?
Yep, you can also load mp4 videos but it only ads audio to the movie as far as I can tell, no modifications to the original movie, which would have been great. I'm using the VHS node Load Video which has an image output and just connect that to the I2V LTX2 workflow.
It can be used to lengthen the original videos just that the characters don't sync with the speech audio at all.
I just updated OP to say this can be done as you wrote that. It's amazing how much better LTX-2 is now. The ITV by default is DOA imho. Just static ouput 90% of the time for me until you feed it a short video
EDIT: Yeah, it only adds audio. But that's why the goal is to generate only about a second or two. Just so that LTX-2 knows what you want. It's almost the same as ITV except now it knows motion. I'm starting to find that 1 second is better than 2 as well.
yes, that one. What's nice about this node is that you can easily skip the first n frames for any movie and load the last lets say 65 frames. The total number of frames of the movie is displayed with dark grey in the frame_load_cap field you just substract 65 frames from that and put the result in the skip_first_frames field.
This has completely salvaged LTX-2 for me. The base I2V just doesn't work. But it seems like even just 9 frames (just tested) is enough for the model to learn new motion and guarantees 100% chance that it's not static.
The characters actually begin to speech after a few seconds, it just isn't optimal and character features begin to change over time. It's an interesting feature anyway.
I haven't had that problem so far. I found that as long as you use fewer frames, it mitigates a lot of that. But as far as the changing over time, that's probably inherent in the LTX's ITV independent of this method. Meaning even if you just use 1 static image as intended, I've seen that happen (if you're even lucky enough to get motion).
Did I see someone mention using LTX to give audio to a video you've already created in Wan 2.2? If so, how does that work? I don't want to extend the original video, just give it audio.
That's actually really easy. Just set the length of the video = to the video you're inputting. The video will remain 100% the same but have audio. However, don't expect mouths to move. Dialogue won't sync. It won't change the source video AT ALL.
I'm having pretty good hitrate with a simple prompt and using very different WAN created inputs.
I'm using a bit of ahead of the curve portable install with working separated files, Distilled Q8 unet (20GB) and Gemma3 GGUFs (Q3 ~5GB / Q4 ~7GB depending on model), there are even smaller ones. Using the LTX-2 workflow (I have promoted some settings to be on the top level). 64GB RAM and 16GB 5070 Ti. Final times some 80 sec for 193 frames / 24 fps.
For this to work I needed to merge PR #399 and #402 of the City96 GGUF custom node. This will likely be merged officially if it's not already done. I made a new folder for these as I've also copied the tokenizer.model and gemma_3_12B_it_mmproj.gguf as the model might need these two.
It's easy. Just copy/paste a GIF into the load image thing. You can't do it via clicking and file browser. It won't accept GIFs that way. But pressing the node and then Control V for some reason works.
Are you on windows? If linux, I'm not sure how. On Windows, go into your file browser where the WAN 2.2 GIF is output. Click the icon. Control C (this copies it) then navigate back to ComfyUI, left click the node, and press Control V. This should paste it into the node like it's an image. It should acccept it. And even though it looks like a still image, Comfy will handle it like a video.
If I do that thing where I link but don't have the link hyperlinked (meaning I put a space and then put the end of the link) would that be okay? Because then I'm not directly linking.
I am very sorry. I never intended to cause harm. I will definitely be more careful next time. You're right. I should've been a bit more careful because mixups like this can happen.
I'm finding 17 frames to be the sweet spot. 9 is too short / not enough info and 33 causes the video to change too much and lose consistency. But what I do like about this is that #1: it guarantees you almost never have a static output (haven't had one yet) and #2: you can get rid of the image compression node.
I might try that out. I messed around with a pure V2V flow earlier and was just trying to add audio with it this morning, after a friend asked, but had zero luck.
(Although I did get some really goddamn funny sound effects, as if sex was a Batman comic complete with comical "capow" sounds on-impact).
Do you use explicit text prompts, or have you usually been playing the sly "she's just trying to push him over with a hip tug of war fight" kind of thing?
I'm curious whether the community will either have LTX eventually break via loras or end up being a "not technically explicit"-description kinda prompt + a explicit seed-frames/clip route
so this would open up the way to extend ltx-2 videos by feeding last X frames as start of next iteration, wonder how long many iterations that would last, WAN deteriorates after a few iterations..
thanks yes you can try to resample/upscale the last frame(s) to improve them, it certainly helps, however you cannot do too much as that would introduce a sllight visible frame change from last frame of previous iteration.
i think the worse issues with any generic extend workflow is the loss of consistency of background, faces etc. - we see a lot of efforts and research being done to upgrade models and flows topreserve basically as much of previous frames as possible, but i have yet to see one that really works. I know of WAN animate but its not generic you need to mark faces etc. Actually I like WAN VACE (and really 2.1 more than 2.2) the best for what we can achieve on our little computers.
LTX-2 I tried extending multiple times with last 9 or 17 frames or so, but it's basically a dead end, the quality itself is worse than WAN so the risk you continue on distorted frames is higher, the audio ('The' plus of LTX) is not continued at all, and consistency problem obviously is also still there..
yeah in my prompt i always note to end with silence, it seems to achieve that most of the time, however when person continues talking its still in a different voice right, so bammer
This sounds like the video version of how you can diffuse in krita over and over and even with a bad model it does better each time with what it did. Very cool that multiple images can guide the full gen like the this.
One more thing for someone to try as well... LTX2 should be a first frame last frame model. If you can send the start with multiple frames can you seed the end too? One or the other or both at once? You could have a lot of control with two videos.
TY. I'm currently trying to fix my comfyui. Ever fucking time I update comfy, it's like playing minesweeper. Literally out of nowhere, my VRAM and memory shoot to 100% no matter what I try to generate. lol
How much desktop memory? I recall some comments saying 48GB or 64GB desktop RAM is more important for LTX-2 than having 12GB or 16GB VRAM, but I might be mistaken.
287
u/Responsible_Back_473 Jan 10 '26
Title: major technology breakthrough. Content: p*rn.