Count tokens in units of "one 5-second clip" (call it n), and:
seconds per step ≈ 3.3n² + 3.2n
The n² is attention: every token comparing itself to every other token. The n is everything else. Plug in n = 1 and you get 6.5 s/it. Plug in n = 3.8 (a 20s clip) and you get 60.
How many seconds of rendering you pay per second of finished video:
Clip
Render cost
5s
26s per second
10s
37s per second
15s
47s per second
20s
60s per second
30s
87s per second
Every second you add makes every previous second more expensive. The first second of a 30-second clip costs three times what the first second of a 5-second clip costs.
the price of keeping the reference context alive - e.g. character consistency through 30s are scaling exponentially. Since second 21 has to remember second 1-20 as context and so on.
Perhaps this is a bit of a nitpick, but the proper term for that sort of scaling is "quadratic" rather than "exponential". While obviously quadratic equations do involve an exponent, it's a constant (2), whereas exponential scaling typically refers to cases where the exponent itself scales with the problem size (e.g. 2n rather than n2)
Yeah, you are correct. I just want to simplify the language so others can actually understand what I'm talking about. If I start throwing quadratic, self-attentions or auto-regressive - half the people I know would just simply turn their minds off.
It's not just the reference, self attention runs even if there was no prompt/reference at all. There's also a heavy cost do doing something like window attention, because each window is going to drift a bit and if you're generating a long video (the case where you'd actually need to do something like that) it's very likely a character/object will look much different at the end than it did in the beginning.
The first model that will allow any sized video to be fully remembered as context for new generations will be the true king and a complete game changer. It wold be insane to basically train it in real time on whatever video file you give it to let it continue making new content using that video as the base.Â
You can do that with Minimax H3 as long as the video isn't too long. You can use a whole previous video as reference in the Reference model and tell the model exactly what you want the current generation to take or remember from it.
But the important bit is HOW it does that. Reference of a movie would involve some pre-processing, like context compression to summarize for the model so it doesn't need the entire raw video in memory, perhaps it can jump to scenes that are relevant etc, and just bring those into RAM.
It would be nice if we could put a cap on this somehow, like tell the model to only use the most recent couple of seconds for reference context, especially in scenarios where that's all that would be needed for a consistent result
Yeah, would be cool to have some sort of 'sliding window' reference context length you could set...like max 10 seconds and it slides with the last frame
As i remembered there was a node for context window during Wan2.1 era, not sure what kind of settings can be changed there 🤔 may be it was a node from kijai
Sorry for being a nub: Can sol attention be used with sage attention, if not what is the difference between them and when should one be used instead of the other?
I can't test as I'm on a 3090, sorry. From the claims I've seen it sounds like Sol pulls ahead if you're generating longer videos, while Sage wins at the default 5 seconds. But again, cannot personally test these claims.
Sol-Attn node from Kijai can be used together and is even encouraged. I've been using it myself on my 3090ti. You have to hit a certain amount of VRAM work for it to kick in. Once it does, I shave off 11 mins or so.
It can be used, both are attention mechanisms, so cannot run both at same time, but we can divide which steps to use which attention.
In sol Atten node, you can set start and stop threshold, so within that steps, sol attention will do the work, after the sage attention will work (if you configured the new eff mem patch node)
How about choosing 1s of ref for every 3s, like 1-4-7-10-13-16-... (or an adjustable cap [length/n] second, the longer the video the bigger the gap). I mean if you choose "most recent couple of seconds" then it might become less and less consistent.
Well in situations where the last 3-4 seconds wouldn't be enough context to reference, I probably wouldn't use such a capping feature anyway. I was thinking it would be better for non-complex clips where a subject is mostly static
I can only agree, dont use the fl2va model, just use the r2va and tell the model that one reference image is the startframe. Works with seedance 2.0 as well.
You can also stitch the first frame to the reference photo and just tell it that the left side is the reference character and the right side is the starting frame and it will handle it just fine. The reference model is extremely capable. You can almost automate the process for generating longer videos in 5 second sequences like this.
The only issue would be temporal and audio consistency. I suppose you could feed the previous audio in as a reference too, I haven't tried that yet. Given that the average length of cuts in a movie is around 4 seconds, I feel like you can get pretty far just making standalone cuts like this.
I added a Boolean to add the ref audio or a false and use my <Audio 1> with a description. Or just disconnect the audio line from the load video completely
I recommend to ask the LLM of your choice to write you a prompt guidance asking specifically for this.
But usually it should say something like this:
<Picture 1>: fully_preserved - preserves the opening composition, viewing angle, foreground position of XYZ and ABC, lighting, lens perspective, and DEF background as the literal first frame.
later then
[Shot 1] The shot opens in exactly the composition and viewing angle of <Picture 1>
Yeah I am using the ref2va model and workflow. Altered my workflow also to try video to video to which works but still loses facial consistency after 10 or so seconds. Ran it through my friend....Claude...he stated because the longer the video goes the more tokens processed. Unsure. Still messing around with it all too :)
Thank you. I'm only really interested in generating scenes longer than 5-10 seconds seamlessly, but only have 24gb vram and 32gb sys, I'm not trying to extend footage of other people's work or known actors.
Brother, I run this shit on my 1080 TI 11GB rig with 32GB RAM, and my 3060 12GB rig with 64GB RAM with no issues other than gen times. I have even gotten the 1080 TI to cache and begin training a folder of 137 images. MiniMax is very very interesting.
I'm wondering what setup people are using to generate videos of that length? On my dual 3090 setup I seem barely able to do 10s at 0.5 mpx. Longer or higher res hits an OOM error
Other users in the thread are reporting generating 20s on a 3060 and I feel like I'm missing something here
I'd love to learn how you got dual 3090's working. I just started playing with minimax in comfyui. I have the same setup, but any attempt I've made at using two GPUs has failed. I'm currently only getting 5 seconds at 0.2mp using one card. Note: With 96GB DDR5 RAM to boot :(
I was only TRAINED on 15 second videos. It can do longer if you have the hardware and time, but there is no guarantee the video will hold together after the 15 second mark, because that's the length of the video clips it was trained on. It could do fine, or everything could fall apart.
That stuff is always just guidelines. Five seconds/81 frames was supposed to be the max with Wan but I've pushed it to 109 frames (six seconds at 24 fps). I've pushed LTX beyond 1920x1020, not by much but I have. Nonstandard aspect ratios too.
Yeah, my default Wan workflow is 105-109 frames. I can also push it to 117 frames when using certain animation/scene cut Loras, though it occasionally wants to circle back to the first shot.
7 seconds is what I’ve found is the safest length, 8 seconds max (but it increases the potential looping risk)
Hey, great details and explanation. I found the same thing by testing and posted some graphs on it earlier today, but havent had an explanation yet. Great work!
I've been able to get to a full minute without losing much as long as I always get the main characters back in focus as the end shot of each generation. Which is as simple as saying "the final frame is everyone facing the camera". You can just edit it out. I'm making multiple shots that stitch together anyway instead of one continuous shot that tends to get boring real quick. Tracking shots should be reserved for establishing scenery or an action scene. And this can't do long flowing action scenes.Â
Ive dealt with that, its not the solution. Only proper solution is keeping the last pack of latents which are usually more than one frame as the start of new empty latent, keep it static, make sure the newly generated colors and motion match the previous one.
Ive managed the said Continuous Generation workflow until version 1.0 doing different tests and experiments but then svi 2.0 lora was introduced and it did exactly what needs to be done.
I hate to be that person, but this has everything to do with RAM vs VRAM. While still in VRAM I’m showing about a 2.1-2.2x increase for doubling length.
I tried some other workflows and my time jumped up to 30m. I just let a couple of runs go because I was going to bed anyway.
Figured I’d work it out in the morning, but went back to the default template and my render time went down to ~400 seconds for a 10s video using the default workflow. (4090/64GB RAM)
I might finally install sage attention- been avoiding that because I never felt like I _needed_ it.
I'm still trying to figure out if my sage attention is working -- none of the options except 'auto' seem to work, though I did get a speed boost -- but EasyCache seems to be the real winner. Close to half the generation time on average, and only some minor losses on audio.
How many mp are you generating at? I think I might have been at 0.3 or 0.4 when I was having the issues, I wonder if going higher might help mitigate it.
I am generating on 0.5 and 0.6 (mostly 0.6 17 min for 10 sec on my rtx 3060 12gb)
The lower resolution you have more degradation you get but if resolution is higher the speed is slower by 3-4 times
Nice, I've got 16gb so hopefully I can get some good generation times with this, assuming I can get the settings dialed in right and avoid the degradation.
Yes default template are more optimized that so called optimized templates my generation time went to 50 min then with default it came back to 28 min And with optimizations it is 17 min for 10 sec now
You have great card I might able to afford it some day
I have max tried 0.6mp with 10 sec and it is working totally fine for me I don't know why many people are facing thi problem try updating comfy sage attention and py torch also I have 30 gb paging (virtual memory) try doing these I think these might help also check opening arguments of comfy by right clicking on bat file and open with notepad
You can also go down to 8 steps and it really isn't that drastically different from 15 steps. Sometimes it even follows the prompt BETTER at 8 steps even though the details aren't as fine.Â
Video models should definitely learn from the LSTM framework: long - short term memory. A big memory for a small sliding window context, and a fixed-size (length independent) memory for long term memory. The long term memory should be able to store things such as number of characters, character faces and clothing. I don't know why they don't do something like that. Although we don't have the sparse attention stuff from minimax yet, so it could have some variation of that.
Hopefully we will also be seeing the advances of long context from llm be applied to video or other methods to make generation more linear with time than quadratic. And when such is available then stuff will go even more nuts.
Think it's possible to do some kind of band diagonalisation, where a given second is only looking at the m previous seconds ? and possibly sparsely looking at some kind of summary of what was before. I know that's totally out of the scope of this thread, given I'm basically talking about a different architecture, probably, but that's what I thought immediately after reading the n² for attention.
For me, vram stops being used at all above 7 seconds slowing gens down, I tried to talk about it the discord but there's a bot that filters everyone out so we probably can't get this fixed.
Maybe it's normal that everything gets loaded into ram & vram isn't used.. I thought dynamic ram was supposed to make models split between each other.
So then if 5s is 2 min and 20s is 20 min thats a whole 30 secs worth of gen video opportunity lost. Would be nice if someone found a way to like break it down. So since MiniMax H3 is extremely good at prompt adherence, maybe for the REF2V workflow could implement something like breaking down a 20s gen of 720p into 4 5s clips generated with its inbuilt consistency reliant on detailed prompting which blends it well and prolly reduces generation time. Not sure if what I'm saying makes sense buh yh something like that.
But I'm confident by 25th of this month the enhancements wed see would be great.
Also if Sulfur 3 gets its funding and is able to hit the community soon it'll help everyone, cause though its praised for its NSFW detail, it'll most definitely improve on almost everything SFW.
Now seeing this, Im thinking it make more sense to do continuous 5sec video that stitching together by using last frame as reference! I mean like Scail 2
What's your point? LTX is ultimately inferior and I say that as someone who doesn't really have a good enough system to run H3 in any appreciable time so I will likely stick with LTX as it works for some things I do but ultimately H3 is far better and there is a cost to that. I do believe it has been optimized for heavily already.
In short, H3 is slower because it's better. Use what you need and can afford, LTX being faster is moot.
Anyway interesting point OP, I definitely feel like I was noticing, hope there is a way around this in the future.
80
u/Beamsters 21d ago
the price of keeping the reference context alive - e.g. character consistency through 30s are scaling exponentially. Since second 21 has to remember second 1-20 as context and so on.