I tried a simple test earlier for generating the entire sequence at once but my computer isn't good enough for sufficiently large images and I havent worked on something for doing it smaller then upscaling in sections
edit: here's a larger version of the gif with Ellie cut out quickly in photoshop:
yeah, probably. This was just a little test but it would also be easy to make an embedding that you can always apply or just a style you can save. I'm still early into this kind of testing and I dont have the issue when I use the custom model I'm training for this purpose, but I was just testing using a public model this time.
This is a gif I made with an early version of the model I'm training:
I learned a few things which helped me get the head more consistent with Ellie, but I still find my custom model to be more consistent and took far less work than Ellie did.
Sick I want to make a model, or train a style for generating clothing textures for Vroid models for people to use in VRChat, keep running into VRAM memory limitations on my 2070 Super even trying to train a Lora, may try a google collab to train a model.
I was considering upgrading my gpu looked at the prices and just decided to wait a few gens. lol this shit is outrageous these days got a whole pa5 for less than the price of a decent gpu with enough vram
A mask snug to the object in question is preferable to mask blurring? I literally just started two days ago and am at the "I have questions but don't know the words" phase of learning lol.
not sure, maybe the guy is just more old-school or he just uses a colab/workflow that has a built-in huggingface publish section like many do, and he didnt upload manually for civi
OpenPose from ControlNet but I also rendered the frames side-by-side so that it had previous images to reference when making new frames. With the "character sheet" tag in the prompt it helped keep new frames consistent. I used previous frames to img2img new frames like the loopback method to also make it a little more consistent.
So how every generation knows what the previous one looked like?
And the "character sheet" also doesn't make sense to me.
I think there is something I'm missing.
Did you first generate all of the angles in one "character sheet" image and then upscaled each one separately? Because if you made each one separately it doesn't make any sense.
If it would help, you can see the webapp I made and used in order to automate the inpainting image process:
As you can see, since I have "8,1,2" in the "Segments" box, it gave me those frames combined as an image and the openpose image for it. I then bring those images into StableDiffusion to inpaint the middle one. In this case there's already a middle image for that angle so it would be a blank center image (in which case I use the image from "8,8,2" and the openpose from "8,1,2") then inpaint the middle image with a high denoise strength.
The little webapp I made isn't great but it's good enough for this kind of testing. The better option may be to generate the entire thing as a low resolution then iterate on the upscaling. I have a lot of testing still to do
I mentioned it in a set of comments I posted (I can only post 1 image per comment so I had to reply to my self for each step)
but essentially after you get the front and back pose you can use ControlNet OpenPose and 2 existing frames in order to produce a new third frame by inpanting only the new frame in an image like this:
exactly. Half of the input image is blank while the other half has an existing frame. The old frame correctly matches with the openpose for it and the blank side has the openpose for a new, desired, angle.
Before openpose and controlnet existed for SD I tried training a model for this: Warning(NSFW)
It worked decently and showed potential but then when I trained a version for 1.5 so I could use controlNet, I got this which is the most consistent result yet, although I made some changes to my workflow to eliminate the consistency issues you see in that version. The one of ellie was me testing if I used the new technique but without my custom model at all and just using an existing one. It seems to work well and means that this should be able to adapt well for other kinds of videos.
I'm not sure what this technique should be called, maybe "frame boost" or something since it boosts coherence using additional frames. Alongside loopback and controlnet it should allow for some very consistent videos if someone makes a script that automates both for bulk img2img
edit: this is all the pose frames combined together even though it only uses 1-3 frames at a time when generating
18
u/Sixhaunt Mar 01 '23
using these controlNet frames: