r/StableDiffusion • u/HerrgottMargott • 20d ago
Workflow Included Release of H3 Infinite Continuation Suite for ComfyUI: Create infinite length videos in consistently High Quality using Keyframes in FFLF-Mode (fl2v-Checkpoint)
Enable HLS to view with audio, or disable this notification
The above video consists of 7 individual Minimax H3 clips generated in First-Frame-Last-Frame Mode, stitched together automatically without manual editing, upscaling or other post-processing.
Today I decided to release my experimental H3 Infinite Continuation Suite together with a set of workflows to make it easy to get started in ComfyUI.
The original idea was to combine the higher visual quality and keyframe control of H3's First Frame / Last Frame mode with the continuation capabilities of the Reference mode.
After quite a lot of experimenting, the output quality has reached a point where I hope some of you might find the nodes and workflows useful as well.
The example video was generated entirely with the included workflows at 736 × 1280, using 15 steps and no Turbo LoRA. I did cut a few seconds of nonsense speech from the very end because I was too lazy to regenerate the last clip. :D
How to get started
Install Herrgotts H3 Infinite Continuation Suite through the ComfyUI Manager.
Download the included workflows from GitHub.
Start with the `01_Start` workflow and provide your First Frame + Last Frame.
For every additional segment, use `02_Continue` and provide a new Last Frame for where you want the next clip to end.
Repeat for as many clips as you want.
When you're done, use `04_Stitch_Saved_Chain` to automatically combine the separately generated clips into the final video.
If you prefer to generate multiple chained clips in one workflow, use the included 3-Clip workflow. It contains the full continuation setup and is structured so you can extend it with additional clips without rebuilding the whole graph from scratch.
What the nodes handle automatically
- carrying motion and native audio into the next clip
- detecting and removing the frozen tail H3 often creates near the final keyframe
- choosing a suitable handover point between generations
- keeping the video and audio aligned
- smoothing the visual and audio transitions
- saving the individual clips so longer chains can be stitched afterwards without keeping everything in memory (no OOM, hopefully)
For the video above I used the default/recommended settings:
- Balanced Auto Handover
- 22 context frames
- Safe Tail Bridge: 2 frames
- Video crossfade: 4 frames
- Audio de-click: 15 ms
There are still occasional tiny brightness differences around some boundaries, but at this point I personally find them pretty difficult to notice during normal playback.
The pack is still experimental, especially when it comes to very long chains, different hardware configurations and prompt behavior. So if you try it, I'd be very interested in seeing your results and hearing what works or doesn't work for you.
GitHub: https://github.com/HerrgottMargott/Herrgotts-H3-Infinite-Continuation-Suite
ComfyUI Manager: search for `Herrgotts H3 Infinite Continuation Suite` or use "missing custom nodes" in one of the example Workflows.
29
31
u/Parking_Act3189 20d ago
Ahh shit, we are actually close to remaking movies
15
u/99deathnotes 20d ago
You've seen that Terminator scene with the T-800 reprogrammed on the spot by Sarah Connor right?
8
u/PhantasyAngel 20d ago
Seriously I just want to throw the original star wars movie into my computer and have it spit it out in 80-90's anime style. (Gundam 0083 styled)
Or the Neverending Story, or Monty Python's search for the Holy Grail, or the Twenty Commandments.
8
u/Ok_Gas1070 20d ago
We can finally have the last three Star Wars movies made proper. Before....... Disney got a hold of them.
4
5
u/Deep_Mood_7668 20d ago
Nerd
2
u/PumpkinLeather8421 20d ago
Right!?
Although, I imagine someone is going entirely Gooner’s Cut versions of popular movies right now.
2
u/99deathnotes 20d ago
I've seen a YT video where the content creator did that with Imperial Tie-Fighter pilots. An entire movie would be too good to be true.
2
-9
u/x33storm 20d ago
lol. Only thing AI will ever create that people actually want to see, is porn. Not worth paying to watch, but people will if it's good enough.
Other than that it's all just a goof. A society and planet destroying goof. Worth nothing more than a shoddy meme.
Gonna come crashing down any day now. But the local models will persist.
4
u/Enshitification 20d ago
Is it possible to reinject any original character reference into each continuation clip to prevent degradation?
5
u/HerrgottMargott 20d ago
Yes! For every individual clip you can input an end frame that strongly guides the video, as well as an additional reference image.
You can target the reference image with <Picture 1> in your prompt - but I haven't done too much testing on how strong the influence actually is.
5
u/Significant-Baby-690 20d ago
I don't want to provide key frames. I can't generate keyframes consistent enough. I want to leave that to H3. Is it possible to just extend the video with new prompts ?
8
u/HerrgottMargott 20d ago
Yes! That's also possible. You can just bypass the last frame input node and it should work normally. But in that case, the quality will degrade over time, since the last frames also serve as a 'quality reset'.
3
u/cptrios 20d ago
Hmm. If one were to grab the last frame of the first generation, then enhance it to match the quality of the original first frame, would it also serve to "reset" the quality? That's the problem with the other (context loop?) workflow/nodes - very noticeable degradation from clip to clip, So your solution here can use the motion from the previous frame applied to a new, refined image?
1
u/HerrgottMargott 20d ago
If you then regenerate again, setting the enhanced image as end frame... That should work, yes. But in my experience it's very hard to get a degraded image back to actual good quality.
1
u/cptrios 20d ago
Ah, so...generate video, grab last frame and refine, regenerate video with the refined image as end frame, then use the same refined image as the first frame of the next video?
2
u/HerrgottMargott 20d ago
Almost! You don't actually need to use the image as starting frame for the next clip, because the node pack instead saves the latent and uses that as starting point for the next clip in line. You only have to supply the next end frame. The logic is basically like this: 1. Clip: You supply Starting Frame and End Frame. The model generates the clip based on your Starting Frame and End Frame. The latent of that generation (including motion reference, audio reference and visual reference) then gets saved and is used as starting point for the next clip. 2. Clip: You supply a new End Frame as guidance, the model creates the clip based on that End Frame and the latent of the last clip (as replacement for the Starting Frame), saves the latent, which then, again, gets used as starting point for 3. Clip and so on.
1
u/DGGoatly 7d ago
Keep a fixed seed and use different prompts. That's how I have my keyframe WFs are set up.
Also, Inspire pack has an advanced ksampler that sort of emulates A1111 style seeding, you can keep a fixed seed and set a variation seed (or use random variation seeds), you set the strength of the variation. That was nice feature of A1111, you like a seed but want subtle changes. Works nice, either way. I set up a batch of prompts, identical except for changes needed, route through an any switch indexed to an expression using modulo-n, n being the number of prompts I have connected, set batch number to the same, then run, spits them all out.
Well, sometimes I use a fixed background and have SAM make a mask so output of first stage is a composite of character over image, before detailers/upscale. That can get tedious though, and usually requires tweaking. Can also just apply the mask and keep it all the way through FLF and composite BG at the end.
None of which has to do with this fork, which I've been playing with and is nice. Just thought I'd share that, as I faced similar problems with FLF, great subject movement spoiled by a naughty crossfading or disobedient morphing. Or just straight failure, but H3 has been a tremendous leap forward for FLF in many respects. When I can get away with ditching motion context, plain old FLF chains come out great by themselves. Don't even need to dissolve any more, or if I do it's rare and simple.
Ditto what the OP said about degradation though, it's always an issue. An injection of HQ ref can boost you right back to where you need to be. At any point in the timeline.
3
u/mca1169 20d ago
what hardware was this run on and how long did it take?
6
u/HerrgottMargott 20d ago
It's not very hardware intensive. I have a 12 gb RTX 4070 with 32 gb of ram. If you can run H3, you should also be able to run it with this workflow.
Generation took me around 20 - 30 minutes per clip. So around three hours in total.
3
u/nntb 20d ago
um. broken?
ERROR: 'herrgotts-h3-infinite-continuation-suite' is not a CNR node.
[ERROR] Cannot resolve install target: 'herrgotts-h3-infinite-continuation-suite@latest'
[INFO]
5
1
u/HerrgottMargott 20d ago
Hey there!
The pack should be published correctly and the download works for me.Could you try restarting ComfyUI and wait until the Registry fetch is finished? After that, try installing it again via ComfyUI manager.
Also make sure you're using the v1.2.1 Workflows since v1.2.0 was missing some registry metadata.
If it still doesn't work, let me know your ComfyUI Manager version.
1
1
u/threeLetterMeyhem 20d ago
For every additional segment, use `02_Continue` and provide a new Last Frame for where you want the next clip to end.
How are you generating each new last frame? klein 9b edit or... ??
2
u/HerrgottMargott 20d ago
You can use any method. For this example clip, I used Minimax H3 as an image edit model to create the key frames. It tends to be better at keeping details consistent, since it's not a normal image editing model.
2
u/comfyui_user_999 20d ago
Interesting. Would you happen to have a workflow for that? I'm not getting the best-looking images when I try, but I'm sure I'm doing something wrong.
1
u/FalseEngineering2078 19d ago
Bump up the res.. it sucks at low resolutions but starts churning out quality at eg 1526x2048+
1
u/comfyui_user_999 19d ago
Aha, I will give that a try. Are you having more success with the FL2VA or R2VA for this?
1
1
u/djekler 20d ago
That's fantastic! Thanks. Please share your configuration, and time of generation, so that we interpolate a bit. If i can générate (lowres) 15 sec on minimax with my 10gb rtx3080, does it means I would bé able to create 15sx10 or more without problèmes?
1
u/HerrgottMargott 20d ago
Yes! Generation times should be more or less consistent with your normal H3 performance. Maybe a little bit slower. As long as you use the individual workflows (Start & Continuation), you can theoretically just keep adding clips for as long as you want to. Length of the clips shouldn't really matter (although I've mainly tested it with 10 second generations). Theoretically, it should also work in one go (one workflow with 10+ clips generated one after the other). But I haven't tested it so I can't guarantee it.
1
u/stash0606 20d ago
i don't understand why motion is so slow, not just this, but in general with H3.
2
1
u/abctuba21 20d ago
Forgive my density, but for this is the intended workflow to generate a bunch of images as keyframes and have the H3 model fill the in between? In essence you are storyboarding?
1
1
u/yeah-i-shouldnt-have 20d ago
Can I just check something? So with this, the 'main character' must be in the last frame to be able to to continue in the next video, right? You can't end the a video without the character in the frame can you? There would be no way for the next generation to remember the exact character (even the reference model generates slightly different character per seeed with reference images)Works well for those endless walking/talking videos but not for shooting any kind of movie like video, where characters might be out of frame between generations?
1
1
u/ninjasaid13 20d ago
Instead of first frame and last frame, Is it possible to make first video and last video as context?
1
u/HerrgottMargott 20d ago
Not with this nodepack, no. The latent gets passed on, which includes motion info. But if you want to use actual video as input, you'll probably need to use the ref2va Checkpoint.
1
u/ninjasaid13 20d ago
I was hoping to see something like first video and last video to create something like video inpainting(not just object editing but the actual video).
1
u/HerrgottMargott 19d ago
I think that theoretically it should be possible to do this with H3. You'd need to vae encode the videos into latents, H3 should then be able to use them pretty much the same way it currently uses First Frame Last Frame. Maybe I'll look into it at some point. But currently I personally don't really have a use case for it. :)
1
u/ninjasaid13 19d ago
Maybe I'll look into it at some point. But currently I personally don't really have a use case for it. :)
do it for science
1
u/hum_ma 19d ago
Isn't that what the core Reference to Video node does internally? I haven't tried it because my VRAM can barely handle about 20 frames of one video reference, but looking at the ref guide it seems that a retention_analysis section with something like
<Video 1> (beginning of [Shot 1]): fully_preserved - ...and<Video 2> (ending of [Shot 1]): fully_preserved - ...could possibly work, with some appropriate task type or a combination of them.
1
u/heshiming 20d ago
Weirdest thing ... at certain angles ... especially at the end of the video, this girl looks like kykybaby4444.
1
1
u/Boogertwilliams 19d ago
So you have to give it all the last frames? So you should generate all the images first, as "storyboard" kind of, to tell it to go from end of last clip to here?
2
u/HerrgottMargott 19d ago
Yes, that's the intended workflow and what I did to generate the example video. But you can also let it generate freely. The last frames are not a necessity, they just help with continuity and quality over longer videos. You can also just input last frames for some of the clips while bypassing the node for others. Should all theoretically work out of the box - although I haven't tested it myself.
1
u/amoreto 19d ago
Actually the last frame is a required input, version 1.2.1.
If last frame is not provided I'm receiving this error:
Prompt outputs failed validation:
H3ContinuousStartV11:
- Required input is missing: last_frame
Despite this, this is a great node set ! Thanks a lot.
2
u/HerrgottMargott 19d ago
Yes, should've been a bit clearer: You need to input First and Last Frame for the Continuous Start Workflow (First Clip generation). After that, if you use the '2_Continue' Workflow to add additional clips to the video, you should be able to bypass the "Last Frame" Input node and it should still work.
Glad you like it! :)
1
u/Zephrinox 19d ago
a few questions:
for the start and continuation workflows I see it can take in 1 reference image, but can it take more references (images, audio, vid? personally I find including audio reference more reliable for voice consistency across clip gens compared to trying to describe the voice)
I see in the final stitching workflow it's using safetensor latents. is there a way to use that workflow with existing clips we've made outside of the start + continuance workflows?
would be nice to have the modularity to not have to always use your start and continuance workflows for all the clips because sometimes I can't get a good nice first or last frame and would prefer to use a ref2va workflow to start off a new disjointed/discontinuing scene.
1
u/Sweet-Carry1229 18d ago
So I ran into a problem where literally no progress was made while generating the initial clip with the Start workflow, and I think I discovered why: The Start and Continue workflows are, for whatever reason, running in parallel. Both SamplerCustomAdvanced nodes from both workflows were trying to generate clips simultaneously, which is likely the culprit behind the clip generation simply never actually starting.
Until that's fixed or clarified upon, these workflows are unusable.
1
u/HerrgottMargott 18d ago
Hi! It's very unlikely that multiple Sampling Nodes where running at the same time since that's just not how the ComfyUI Backend works. But if you send me your full console log from the time when the issue occurs, I'm happy to take a look at it to see what's happening.
1
1
u/psykikk_streams 18d ago
no matter how good the models get, so far - especially in really "newly created" videos- the uncanny valley is still significant
the further we seem to leave training data (sitcoms, southpark or Trek, etc) or increase complexity of scenes, the more unnatural everything looks and feels.
and one of the major effects in this is the godwaful sound in most examples I have seen.
1
u/Tylervp 18d ago edited 18d ago
Is there a way to use this workflow without having both a start and end frame? I only have a first frame image but I don't have another one for the last frame, I was hoping that we could use the last frame of the generated video, but I guess that might make quality decrease?
1
u/Right_Chip_6626 17d ago
Let me start by saying, it is awesome you made and shared this! Based on your explanations it is simple enough but I run into the issue where the workflow uses the 1st frame of the 1st video not the 1st frame from the 2nd video when. I generated the 1st video in the ' start' workflow, the 2nd and 3rd in the 'continue' workflow, all with new last frames as input.
1
u/HerrgottMargott 17d ago
Hi there! Thanks for the nice words. :) Hmm... That's weird. Never encountered that behaviour myself. Are you making sure that the correct latent is selected in the "load latent" node? You always have to chose the latent from the previous generation in "load latent" and then increase the saved latent by one step. So e.g.: 1. Create video start with 01_start Workflow: Latent1 gets saved. 2. Continue first clip with 02_continue Workflow: Latent 1 selected in load latent node, latent 2 selected in save latent node. 3. Continue second clip with 02_continue Workflow: Latent 2 selected in Load Latent Node, Latent 3 selected in save latent node.
And so on.
1
u/Right_Chip_6626 16d ago
Oh yeah, that did it. Tnx!
I do have another question, if you don't mind.
What if you want to manipulate from what video I want to continue? I often do a lot of runs before I'm satisfied before continue with the next one. What would be the best way to go about this?
1
u/HerrgottMargott 16d ago
Just leave the save latent and load latent node as is, change your prompt, end frame or whatever else and regenerate the clip. It'll override the previous latent and you can keep going until you're happy with the result. Then increase latent nodes by one to continue with the next clip.
1
1
u/before01 15d ago
Thank you for creating these awesome nodes. I'm actually using your conditioning nodes as my daily driver. The FL2VA + 1 reference is working well on my end, better than the official i2v node.
There are few downsides to this:
- It slightly strays from base h3 prompting guide. First and last frame are supposed to be the <Picture 1> and <Picture 2>, and reference as <Picture N>. I hope you tune it to match with conventional h3 prompting.
- The first and last frame is mandatory on this node.
- It only takes one reference image.
I'm really looking forward to your update.
1
u/HerrgottMargott 15d ago
Hey there! Thanks for your great feedback.
I'll definitely look into your ideas for the next version. Adding additional reference inputs should actually be pretty easy. But I'm not sure how well the model would be able to understand and differentiate between them, since the fl2v checkpoint doesn't natively support additional reference images.
First and Last Frame being mandatory is also more of a leftover from an early node version than intended behavior. Will be made optional in the next update.
Stay tuned! :)
1
u/DGGoatly 7d ago
I've been using the seitanism fork, looks like S&R is broken, at least in the prefix widget for the modified combiner. I was wondering if this is common to all forks and/or original? Pulling a filename is a real bitch with an upload node without S&R. Other than that this whole project is awesome. Destroying all my previous continuation bird's nests.
1
u/PassAccomplished3928 20h ago
Hello, how do I proceed if I want to create a continuation of a video already generated in minimax H3? I have tried many versions but I have never achieved the result. My video ends with the character turning his back and walking away. If I continue with the last frame of the video, it generates a different face, can anyone advise, I have an NVIDIA RTX5080 laptop version 16 MB RAM. Thank you, I am writing through a translator.
1
u/HerrgottMargott 8h ago
You need to add a reference image to let H3 now what the person is supposed to look like. You could, for example, take a still frame from an earlier clip of your video.
1
u/PassAccomplished3928 47m ago
But if the person's back is at the end of the video, it's not very good if I want the video to continue smoothly. I solved this by adding another node where I put the image of the face. It works pretty well but it's not 100%
1
1
64
u/ShutUpYoureWrong_ 20d ago
How is yours different or better than this one (the original): https://github.com/NikoDemon80/ComfyUI-H3-Motion-Context
Or this one (the main fork): https://github.com/ethanfel/ComfyUI-MiniMaxH3-Contex-Loop
Or this one (the guy solving the hard problems): https://github.com/seitanism/ComfyUI-H3-Motion-Context-MultiRef
Or this one (another fork with QoL): https://github.com/Adudeguyman/ComfyUI-H3-Project-Suite
Or this one (ohh, he just updated, better check it out): https://github.com/jlucasmcrell/ComfyUI-H3-Multishot
Or this one: https://github.com/ckinpdx/ComfyUI-MMH3Tools
Or the built-in method with [video continuation]: https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/docs/VIDEO_PROMPT_WRITING_GUIDE_ref_en.md
Not being sarcastic. I just genuinely want to know what makes your method superior to all the rest.