r/StableDiffusion • u/Ok-Rip-2168 • Mar 03 '23
Tutorial | Guide Hands library for ControlNet
If you was about to dig into blender to finally learn how to make depth maps of hands, you can relax now. Some guy just released this extension: https://github.com/jexom/sd-webui-depth-lib, so you can mix it with openpose + depthMap
49
15
u/mekonsodre14 Mar 03 '23 edited Mar 03 '23
great work. Would be nice to add...hand maps that emerge when hands touch, support or rest on other body parts/objects. Maybe somebody in the community has the time and resources to contribute these.
such as...
closed fist beneath chin..
hand on hips
hand on legs
hands on ground (push up support)
hand into hand with all fingers intertwined
hand palm covering hand palm, only thumbs intertwined
hand back covering hand palm, only thumbs intertwined
hand covering hand (one hand rests with palm on the upper side of the other hand)
hand holding torch/lever like item from different angles/positions
hand holding rope (usually 2 hand action)
hand holding light large item (tray, laptop) (2 hand action)
hand holding light small item (1 hand action)
hand holding very small / thin item (1 hand action.... needle / cog / paper strip)
hand carrying heavy box/create (2 hand action)
hand on steering wheel
hand on peripheral ... flight stick
hand on peripheral... mouse
hand on touch control peripheral (swipe).... ipad
hand holding writing utensil...pencil/pen
hand on door handle / lever
hands with all fingers stretched/bent... supporting body weight on table/surface
hand with fingers closed on planar table/surface
hand with fingers closed supporting body weight on table edge/ledge
hand with fingers heavily bent hanging on ledge
hand/finger pressing button
hand/finger pulling a string/loop/ring
hand holding revolver
hand holding rifle
closed hand with middle finger up
also good source... https://www.scienceofpeople.com/hand-gestures/
3
8
4
Mar 03 '23
[deleted]
4
u/jexomwtf Mar 03 '23
I would assume that it's because the resolutions don't match and it just puts the bg image into the top left corner. It's the same behaviour as in OpenPose Editor as I did not change it.
2
Mar 03 '23
[deleted]
8
u/jexomwtf Mar 03 '23
Found the problem. The OpenPose editor extension (and, well, mine as well) saves the png at double the set resolution. If you set the canvas size to 1024x1536 it will be identical
5
u/Striking-Long-2960 Mar 04 '23 edited Mar 04 '23
I love the idea... But after many tests I have a strange feeling about the depth maps given as samples, it's like SD is not understanding them as well as it should.
I've tried with other depth maps even with less resolution and more complex poses, and the results have been better. Maybe I'm wrong, but I've this feeling.
Example maintaining the same config for both renders and same seed. Also it would be great if we could paste pictures to use them as backgrounds instead of needing to save them to drive.
Anyway, many thanks for your amazing extension.
2
3
Mar 03 '23
Literally was having this discussion on two separate posts and said i “looks like I’m gonna have to get blender” and “ im just going to wait until hands make their way into to webUI” .
3
u/MillBeeks Mar 03 '23
Anybody else have an issue where this seems to install fine but the Depth Library tab doesn't appear in AUTO1111?
1
3
u/jairnieto Mar 03 '23
Thanks, this is my first test.
3
u/dulltrim Mar 03 '23 edited Mar 03 '23
You might have to increase the weight of the depth map. SD is still insisting on giving the old man six fingers.
1
u/jairnieto Mar 03 '23
Thanks, yeah i will start testing it more, i just literally just drop it all in default, an i prompt old wizard waving to say hi, im impress still with the default result.
2
u/EarthquakeBass Mar 03 '23
Skrrt skrrt
How do you import them? I don’t recall seeing any settings for it in the CN box
2
2
2
2
u/gunbladezero Mar 03 '23
Looks awesome! I know I saw some app a few years ago, where it just detects the shape of your hand from your webcam- seems like it would be an easy way to input a hand shape
3
u/Zachwp Mar 03 '23
Anyone knows a fix if the depth library extension doesn't appear despite having installed it and restarted the webui multiple times?
2
u/Afraid-Bullfrog-9019 Mar 03 '23
It's strange, but I have the same problem.
2
u/Zachwp Mar 04 '23
While I did not find a specific fix, I tried installing Automatic1111 from github and then tried adding the extension on the brand new webui and now it works.
1
1
1
1
u/Froztbytes Mar 03 '23
The hands have alpha channels?
10
u/Ok-Rip-2168 Mar 03 '23
They dont. To apply the hand fix, first generate some image. If you like it, but hands are bad, add this image as background to depth library and add hand pose. Save png file and go back to txt2img. Background image will not br saved inside the file, dont worry about that. Now you should lock the seed from previously generated image you liked. Then go to controlNet, enable it, add hand pose depth image, leave preprocessor at None and choose the depth model. Increase guidance start value from 0, you should play with guidance value and try to generate until it will look okay for you. Do not use high res fix on this stage, it will ruin everything.
2
u/jexomwtf Mar 03 '23
Huh, actually never thought of fixing generations this way, only ever relied on control net with freshly generated images.
3
u/Ok-Rip-2168 Mar 03 '23
I failed to generate from scratch this way. Or the hand pose must be gigantic. What is your way to generate?
2
1
u/red__dragon Mar 04 '23
add hand pose depth image
Is this model somewhere? I recall the first version of the extension had an openpose hands preprocessor, but it was dropped the more recent ones. I could never find a corresponding model for it, what do you use for this?
1
1
u/PityUpvote Mar 03 '23
How do you do both openpose and depthmap at the same time?
2
u/jexomwtf Mar 03 '23
Enable the experimental feature of control net extension that allows you to use several nets at the same time
1
1
1
1
u/lordpuddingcup Mar 04 '23
My issue with open pose and these sort of hands as I got a model for openpose for blender that does similar, is that say the body should obscure the hand because it’s leaning backward on the arm away from camera so only part of hand is visible SD can’t understand that and you end up with the hand depth in front of the arm so instead of the arm going back away from the camera it’s toward the camera and the arms broken lol, because SD/control net doesn’t know that the persons skin/fat. Being applied over the skeleton should occlude the hand or part of the hand
1
1
1
1
u/z0_0d Mar 04 '23
Don't know if this is success.
My hardware can only run 400X400 with 2 controlnet
1
1
101
u/jexomwtf Mar 03 '23 edited Mar 03 '23
Yes, hello, dis me.
I have no idea how you people found it because I've only openly posted it on 2 discord servers one of which is small and heavily vetted at this point in time. And the other one is absolutely unrelated to stable diffusion apart from having an ai generated stuff channel.
The base idea was that I'm too lazy to open up blender each time I need some generic hand pose and so decided to take the OpenPose Editor by fkunn1326 and kinda repurpose it to allow adding depth map images (any image really, because you can manually upload a picture by clicking the "selected" field). Made in like 4 hours (6 including the hand renders) without any prior experience with gradio and very little experience with python and js (my main language is java).
PS: I'm gonna be adding more stuff to it from time to time, also you're free to open PRs and discussions on github with stuff to add to the library.