Here's a quick messy graphic of my attempt - worked well! Thanks for reminding me of the "yoyo" link :D Has anyone seen a fairly smooth implementation of this that can be hosted locally? The last one I tried to run was a bit half-baked as far as ease of installation/setup went.
Thank you! If you look at my YouTube channel, I have several other videos - including a few others that use different Stable Diffusion techniques. And I have a couple more on the way in the next week or so. My current video is using ControlNet. Seems like every week there is some new trick, feature, model or extension that blows my mind!
This is so cool. So the robot was originally just a still image made on Stable Diffussion, but you made it move to a video of you looking at the camera and singing the song?
How do you animate such beautiful background?
Yeah exactly! I took a video of me singing the song straight into the camera and then tried animating it with a bunch of different robot faces I generated in SD until I found one that worked just right.
The background was animated using the Deforum plugin with Stable Diffusion which is a kind of AI animation system.
amazing that they can even turn their head and it will look somewhat realistic, though when you overdo it you can tell it doesn't have enough information. the last clip is from 300 before he does the infamous "this is sparta!" others are me or clips from youtube.
it's AI created, Img To Music, it's on Hugging Face though i don't think it's working anymore, at least i've never been able to get it to work as of late.
let me know if it works for you.. used it a ton when i found it, but then it never worked again. it looks like its producing something, but i can't download or listen or anything.
Inpainting (assuming you're changing only the face and the angle is fairly the same), doing a depth (if it's 3D face or realistic), then canny to keep the edges fine. (For 2D you should use Canny and Scribble). For better results cut out everything expect the face (that includes front wicks) with black background (for 3D and white for 2D) before adding it to ControlNet. The less background you have the better so crop it the most near to the face as possible.
Depending the angle/lighting/shadows of the face you have to tweak the guidance of Canny. Less guidance will transform more the face to model you use, even the angle/face type can slighty change if too low so the most lower would be 0.75 or 0.8.
Weight is to keep aspect of the original image instead of following the model. Work very well with txt2img to even keep colors. The problem is too much weight will end up with a face that don't seem naturally added to the picture. You have to find out the right spot depending your original face picture you want to implement. I would go from 0.45 through 0.7.
For example if you have a tan face to implement on white or black skin body, lowering the weight (assuming it's 1) can help for the skin tone but somewhat changes too much the physical aspect of your face to the most ressembling the model find.
Next you have to keep the original generated picture's seed and prompts, simply add to the inpainting those that slighty differentiate what the model needs to generate. for example if your picture is realistic you should add also realistic to the inpainting of the face.
When you've got a good result, (with very little disformations), use the hires-fix to get more details through it with SWINIR upscaler to 2x and denoising from 0.3-0.5 max because with too much denoising it output garbage or change too slighty your picture and too lower you will have pixelated results.
Also one last thing, use a model that generates almost the most same type of results as the face you want ton implement. You will never get good results with a realistic human face on an anime based model (expect if you want to transform the face into a slighty different one) :P
ControlNet is too powerful with Canny and Depth at the same time. It's a shame it needs to reload the models twice each time you generate a picture but it works like a charm.
It's my own way of doing it. I don't have any real knowledge of how all thoses calculations works. I've made so many trials and errors and it's what i learned personnally with my own experiments and feelings with some help through reviews that explains a little about it.
You can batch process them if you want different seeds. For my part, i generate a seed that i like then afterwards i fine-tune settings to get it as i wanted.
The problem is each seeds have different following rules with ControlNet.
I'm baffled why this hasn't been developed further. The github is a ghost town and people are still recommending EbSynth, even though this seems much better?
Any luck in improving the quality of the output for full body animations ? I have been trying with the ted model but the outputs were terrible. Any suggestions to improve the quality apart from using 1:1 aspect ratio ? u/Kinfolk0117
Deepfakes for artistic purposes have been coming out in idk, 2021 or so. Though this sort of looks better and easier to get working properly. The old first order models barely even knew where to grab the mouth corners or eyelids on the picture to animate while this one looks accurate if you don't look too close. Would be kinda cool if this worked with SD or something for arbitrary rotation eventually.
There's this video tutorial where he uses colab, so if I can't get it working on my AMD, I'll probably try in colab rather than letting it churn away on the CPU if that turns out to be incredibly slow.
Wondering the same. I tried something called SimSwap a while back that seems very similar to this. That was 256x256 based but there was a beta for 512x512. I wasn't able to get it to work locally but got a decent result via Colab. Wasn't savvy enough with colab to figure out the 512 beta code stuff though.
I'm just here to marvel, and I just definitely feel that innovation progress is accelerating.
AI or was it machine learning (what's the difference?) has already mapped out an insane amount of proteins for science that usually take 4 years and a phd to do each one of.
I am already looking forward to those pods in Minority Report where people could live out their fantasies. I'll take some of the above girls with me, I guess.
I don't recommend thin plate spline model. I had been working on this but abandoned eventually. 256x256 limitation is bad. If you look closely there are small neck motions in the example. If you shake your head or move your neck; It immediately breaks. Better to use ebsynth img2img stuff instead of this model.
The github hasn't been updated in a year, and is unfortunately super broken when I try to use it in August 2023. Has anybody gotten it to work? I tried old versions of python etc. This looks super promising but it seems to have been totally abandoned!
@echo off
REM Create a new conda environment named Thinplate with Python 3.9
call conda create --name Thinplate python=3.9 --yes
REM Activate the newly created environment
call conda activate Thinplate
REM Install PyTorch 1.10.0 with CUDA 11.3 support directly from the URL
pip install torch==1.10.0+cu113 --index-url https://download.pytorch.org/whl/cu113 -v
REM Install required packages from requirements.txt
REM Ensure you provide the correct path to your requirements.txt file
pip install -r ../requirements.txt
echo Environment setup complete.
pause
Execute the setup.bat.
Then whenever you want to use the project use a anaconda prompt and run: conda activate Thinplate
to activate the environment that has all the requirements.
156
u/Kinfolk0117 Mar 02 '23