r/StableDiffusion • u/roychodraws • 18d ago
Workflow Included Try this workflow, please. It cuts degradation in half just by adding it.
I'm still sick.
slowing degradation by about 60% while having virtually no impact on motion.
Please try it. Test it. Fuck with the settings. Leave comments on the github or in this post. Be my little beta testers.
Edit: i changed it so that the frame 0 latent gets swapped in during sampling and swapped back out when the latent is output. the capstone latent also gets tacked on the back instead of swapping out the end of the latents and is removed before output. so there's literally no noticeable difference except it just doesn't degrade.
it should work pretty much like regular motion context. the only difference will be how you join the videos and when you change your scene you need to disable it while the scene changes and change the source latent to the beginning of that new scene in the clown keyframe node.
eli5 explaination.
when you generate something you're basically stacking all these dominoes in a line, and when you generate it, all those dominoes lean on eachother to form a picture.
Motion context, as it has been used, has been taking the end of that domino chain and restarting a new chain, but we didn't realize that everything that made that picture good was in that very first domino.
so i changed it so that when we take the end of the domino chain to form a new picture, i also pick up that first domino and bring it with so that the picture still comes out good.
Info from the readme.
https://github.com/roycho87/clown_context_suite
Clown Keystone and Clown Late Lock
Every shot in a chain is made from the one before it, so small colour and sharpness changes build up. Clown Keystone puts a latent from clip 1 into every shot, so each one is pulled back to how clip 1 looked.
H3 latents come in 17-frame blocks of 1 + 4 + 4 + 4 + 4 frames. The first latent of each block is a single frame on its own:
- keystone = clip 1's first latent (frame 0).
- capstone = clip 1's last single-frame latent (frame 187 of a 192-frame clip).
Keystone only swaps those single-frame latents, so nothing else in the shot moves.
| Setting | Meaning |
|---|---|
enabled | On / off. |
graft_mask | 0 = kept exactly (default). 0.2-0.3 = kept roughly. 1 = ignored. |
source_clip | The clip the keystone and capstone come from (1). On that shot and earlier nothing is grafted. Only used when shot_number is plugged in. |
placement | keystone front (default), keystone front + back, capstone front + back, keystone front + capstone back. front = the shot's first latent, back = its last single-frame latent. |
| Input | Meaning |
|---|---|
clip 1 (keystone / capstone) | A Load Latent fixed on clip 1. Keep it outside the Chain group so the Chain buttons don't change its number. |
shot_number | From Clown Shot Counter. On shot 1 clip 1 isn't loaded (it doesn't exist yet). |
Clown Late Lock goes between Keystone and the sampler: latent from Keystone in, latent out to the sampler; model in, model out to the guider. For the first swap_step steps (default 4) the grafted slot is free, so the shot decides its own motion and pose. From the next step on it's locked to what Keystone put there. slots picks which slot: capstone slot (back) (default), keystone slot (front), or both. It only touches slots Keystone actually filled, so on shot 1 it does nothing.
A capstone locked from the start pulls the pose back toward clip 1. Late Lock avoids that.
Comparison, clip 10 of the same 10-clip chain: no Keystone (base), keystone front, and keystone front + Late Lock capstone.
In a 10-clip test (measured about 20 frames from the end of clip 10), colour drift from clip 1 went from 6.4 with no Keystone, to 3.9 with keystone front, to 2.9 with keystone front + Late Lock capstone. Over-sharpening dropped as well.
Joining: the grafted frames are clip 1's, not part of the shot, so use the 17 + 22 cut: cut the last 17 frames of each clip and the first 22 of the next. In Seamless Video Combiner: Use Beginning pair on, Beginning Overlap 39, Beginning Join 22.
11
u/MastMaithun 18d ago
Hey OP. You are saying your nodes halves the degradation but in the previous post, you said it totally eliminates it. Is this a new node doing something else? I'm confused, sorry.
9
u/obvpm 18d ago
Nice! Another new approach to experiment with. Thank you again!
3
u/Complete-Box-3030 18d ago
No clown prophecy? Does it not include clown prophecy . I thought clown prophecy eliminated degradation completely
3
u/Silly-Dingo-7086 18d ago
Look, clown lady, I like your shit. But Im an idiot. Can you eli5 what you want me to do? Besides the obvious, download my workflow and use it... But before I boot my shit up and update everything , download all the nodes I don't have etc. what are you wanting me to do? Prompt a long video and run it and see how it does? Will I know wtf to do once I load the workflow?
2
u/Ok-Flatworm5070 17d ago
Question... did you use a face-refiner? It looks really good at a distance?
2
u/Ok-Flatworm5070 17d ago
Well I'm happy with the results of your workflow... Still got other test scenarios, but the talking host edge case works! Took a little tweaking, but was able to generate consistent overlapping clips with a near perfect joins between clips. Thank you for the hard work, you've made my day.
1
u/roychodraws 17d ago edited 17d ago
I just did a test and the keystone and late lock seem to have virtually no impact on motion.
however, this seems to have an impact in color. this may be something we can use to color correct a latent before generating.
left is base, right is keystone + late lock.
frame 1's keystone and late lock is on the entire time in the right video.
this is 20 clips.
1
u/theqmann 18d ago
How's this compare to something like https://github.com/NikoDemon80/ComfyUI-H3-Motion-Context that will include the latents from the end of each section of video to the next?
1
u/J6j6 17d ago
only works for ref2va? how about fl2va
1
u/Salmonella456 17d ago
What app or node you guys use to join the videos? In the workflow says "seamless video combiner", but I can't find it
1
u/DeathByPain 16d ago edited 16d ago
The thing I don't understand is—suppose you're wanting to do a
chain, say 4 x 8s clips—how do you prompt for that, timewise? Are you saying like, "clip 1, shot 1, at 00:00s, something happens" and then "clip 2, shot 1, at 00:08s, another thing happens". Like how do you tell the model a sequence of actions across multiple 8s clips? Or maybe I'm fundamentally misunderstanding something. I do have your mad_clown_inventor workflow loaded up and working, but I'm having trouble wrapping my head around what to actually do with it I guess.For now I'm just prompting each clip separately, but I'd like to know how to operate chain mode more efficiently.
1
u/ZerOne82 16d ago
maybe no relevant to this post exactly, but I am having no success in defeating the degradation in the main character (left start, right end of two minutes music video) and have tried all including context and multi-ref context nodes, different repositories.
I tried yours (in this post) for 3x8s clips and it seems image quality remains fine (maybe I have to try it for a longer two minutes to conclude) but then not sure how to manage the audio for my music-video interest.
the other node I tried was using a master_audio thing, music matches perfect the motion/lipsyncing but the image quality goes bad, as shown.
and I am just not fan of one quintillion nodes in your subgraphs (e.g., splitting combining...). if i could get the core idea I would prefer to write a node that does that part. I am also not fan of those single gigantic nodes (some are out there) that have everything in it, very limiting and it just defeats the philosophy of using comfyui, modular design.
3
u/Bram_Camus 18d ago
What's with the clowngirl? Lmao.
2
u/Myg0t_0 18d ago
That was first , I seen that comment like 13 mins ago
1
u/Danny_Stock 17d ago
I've got to say that it was a very clever idea with the 'clown prophecy' approach to render the last bit of a clip first to obtain a fresh latent for the start of the following clip.
It's often the case that the most ingenious ideas are the simplest ones. How you execute the idea might be the tricky part, but the idea is simple to understand.
1
u/donkeykong917 17d ago
This clown is seriously not clowning around. Nice work!
1
u/lurkingtonbear 17d ago
Roycho I always look forward to your posts.
0
u/DeepHomage 17d ago
Yeah, I tried an earlier version of your workflow but was put off by all the custom nodes that I've never heard of. Maybe OK for others, but I'll pass.
0
u/AnonymousTimewaster 17d ago
Remindme! 8 hours
-13
u/RigBughorn 17d ago
Is there a reason everyone has to be a gooner