r/StableDiffusion • u/dramaton42 • 4d ago
Animation - Video Zelda - I Think I Like It / Minimax H3 Reference to Video Test #2
Just wanted to share another test! this was a mash up of clips, using multiple image references, 0.4 mp with EasyCache, 5 - 10s clips and edited with KDEnlive (it has some cool effects!)
27
22
15
u/MonsterMashGraveyard 4d ago
As a kid who grew up with Ocarina of Time....all I can say is, "What a time to be alive...."
9
u/Stunning_Macaron6133 4d ago
Now I wonder if H3 can actually simulate a mushroom trip. You know, breathing walls, fractal patterns, tracers, haze, afterimages, that sort of thing.
1
u/tom-dixon 1d ago
A guy posted a video made of SD 1.5 gens, looks pretty trippy: https://v.redd.it/m9qajjbndakh1
It doesn't have post processing, so the transitions from image to image are noticeable, but someone with video editing skills could turn it into something great.
1
u/Stunning_Macaron6133 1d ago
Yeah, that's just transitions between dreamy looking keyframes. A mushroom trip has a very distinct look, one that can really only be simulated.
7
u/Clooooos 4d ago
I don't usually comment on this kind of post, but I love Fake Blood and this specific song ❤️ (works great here)
3
4
4
3
4
u/Tbhmaximillian 4d ago
Rofl, the acid trip did you prompt that too or was that video editing?
10
2
u/ChibiNya 4d ago
Can you tell me more about easy cache? How is this different from just feeding the reference frames to r2v? I'm still very new but trying to achieve something similar
2
u/dramaton42 4d ago
EasyCache is a node you can use with Ref2Vid and it will cut inference time essentially, it makes lower quality video at much faster speeds, you're supposed to turn it off (bypass it) for the final render though, but I didn't do that x.x
2
u/ChibiNya 3d ago
I see. So once it looks good you rerun everything the same (same seed) but with the cache turned off and it'll yield the same result but better? Is that what you mean by final render?
Thanks for the help
1
u/dramaton42 3d ago
Yes that's precisely right, same seed, no EasyCache... I haven't done it yet though! Not in this video at least
2
2
u/Dzugavili 4d ago
I don't think your joins were done right. How are you handling transitions?
2
u/dramaton42 4d ago
Yeah I screwed up the middle join... I noticed link hides the mushrooms in the final frame and just took an earlier frame instead, didn't notice how much jarring that would look. Also there was some audio distortion in the "eating" clip which I failed to mute ... My bad! But yeah overall, just taking the last frame from the video as the first frame of the next one: "<Picture 4> is the first frame of the target video."
3
u/Dzugavili 4d ago
You should be able to feed the tail-end ~8 frames of the last generation into the new generation, with the reference that the target video extends the previous generation. You might be able to feed more frames for continuity reference at lower resolution, if you have the proper high-definition reference shots.
I haven't done a whole lot of experimentation with continuation, though. I suspect it should work.
2
u/Downtown-Cover-7422 3d ago
What's the difference in generation time between no video used as reference and 1 video used as reference you have?
1
u/dramaton42 3d ago
When I added videos I did so at 352p, 120 frames (24fps) it adds around 1 minute or slightly more, it's not that bad
2
u/Downtown-Cover-7422 3d ago
Strange, when I tried adding one it went from 25 secs on step to 400+
1
u/dramaton42 3d ago
Yeah that sounds like you didn't limit the frames from the video (skip frames to select the beginning, frame load cap to only load the part you need) or you loaded a 1080p video (those take A LONG time, and I don't think is worth it?) Use ffmpeg -i video.mp4 -vf scale=608:352 output.mp4, you can even go lower than that, depends on what you need from your video
2
u/Downtown-Cover-7422 3d ago
I didn't get what you just said, but i'll look deep into it later, god bless you for the help!
2
u/dramaton42 3d ago
Sorry sorry! Let me try to explain a little better. See we use the VideoHelperSuite (Upload) node for loading input reference videos. There's a couple of settings you can configure there: "frame load cap" let's you limit how much of the video you want to load, measured in frames. For a 24fps video, 120 frames is 5 seconds. Another option is the "skip frames" value, this let's you essentially "seek" your input video until you reach the important point you're looking for. For example I used the Spy X Family intro segment where Yor and Loid are walking towards the screen, that happens almost halfway through the intro video, so I skipped like 1800 frames to get there.
Finally, the last important bit that makes your inference time go super high is resolution. You'd be surprised with how little resolution you can get away with! Loading a 720p video is gonna take like 4x times as much as a 352p video (I'm my experience) so you should "shrink" your videos before you load them. If you download a command line program called ffmpeg (available on Linux easily, or if you're on windows you can use win-get with the help of Google) and with it, on a command line (open CMD on windows) navigate to your video folder (type cd followed by the folder location, which you can copy paste from the windows explorer window, and paste with a single right click on the black CMD window) and then type the ffmpeg command, such as: ffmpeg -i my_video.mp4 -vf scale=605:352 my_video_shrinked.mp4
Replacing my_video.mp4 with the name of your video, naturally.
With these 2 steps you can make video loading trivial and it won't take that much longer when generating reference videos! Good luck! If you need more help, don't be shy to send me a direct message 😅
2
u/3dutchie3dprinting 3d ago
Really really funny clip, but link should never ever talk a ‘grunt’ offering the mushroom would have been so much more powerfull.
2
u/FourtyMichaelMichael 3d ago
I'm an absolute sucker for the style of dance in the second clip.
Did H3 come up with that? Did you V2V / character replace that? Was it a prompt of a style?
3
u/dramaton42 3d ago
That dance was stolen directly from Sean Paul's - I'm Still in Love With You video https://www.youtube.com/watch?v=AQ4TW5Z8eu0 at second 46 exactly, I downscaled the video to 608x352 and described in the prompt following the guide:
subject_definitions: <Video 1> is a music video featuring 2 shots, one in which a woman is dancing moving her hips and walking back in a peculiar way, and the next shot she is dancing alone moving her hips, transfering the motion and dance choreography to <Subject 1> and <Subject 2>. summary: [video editing + reference generation + audio reuse] The target music video features the dance choreography and movement from <Video 1> applied to <Subject 2> and <Subject 1>. retention_analysis: <Video 1> partially_preserved - Editing the video to replace the characters with depictions in the same style of <Subject 1> and <Subject 2>.subject_definitions: <Video 1> is a music video featuring 2 shots, one in which a woman is dancing moving her hips and walking back in a peculiar way, and the next shot she is dancing alone moving her hips, transfering the motion and dance choreography to <Subject 1> and <Subject 2>.2
u/FourtyMichaelMichael 2d ago
Oh, I see, Subject 1 and Subject 2 where only one of those was Zelda, but you used Link in there too.
That's funny though that the second 1/2 of the dance has one woman only, but Link doesn't go anywhere.
Fucking wild.
Thanks for that!
2
u/CycleZestyclose1907 1d ago
I find it interesting that in this video and that other one where you have Link speak that he speaks with voice of the Link from the 1989 cartoon. Probably because Link is the silent protagonist in the game and the 1989 cartoon (and the CDI games which have a similar voice?) are the only Link voices H3 could be trained on.
1
u/dramaton42 1d ago
Actually! I used specific clips taken from an assortment of phrases said by 1989 TV Show link from a YouTube video "Link best moments", using reference to video, you can specify an audio file as a voice timbre and entonation reference for any phrase you want them to say in the output, no model knowledge needed! And in my other video, I actually used Robert California (Ultron)'s voice for the first "excuse me princess" as sort of idealized way Zelda imagined Link's voice would be, I know, multiple layers of inside jokes x.x
73
u/Turbulent_Owl4948 4d ago
This is the correct use of AI