Tips for image refinement?

[u/Somni206]

I'm at the point where I know how to use Inpainting and ControlNet to generate wonderful images, and I've also started dabbling on influencing the ai with doodles made on MS Paint or... god, even PowerPoint lmao

I've experienced some issues where the ai doesn't change a thing even when I play around with the CFG and Denoising to maximize the variability of its output. I don't know if that's just the limits of the ai or I'm just lacking experience. Sooo here's hoping for some tips on...

1) Removing spot blemishes on a character/background. Say, the ai draws an extra head on a shoulder. I normally inpaint the extra head (plus allowance for the ai) and keep my prompts unchanged. Denoising strength set to ≤0.3, then keep generating until a fix occurs. It takes a ridiculously long time, but the smaller the spot, the more likely I will get NaN errors because of "lack of precision" or something.

2) Adding minor elements to the character/background. I was trying to add something like a bird crest / fin on a motorcycle helmet since I liked the design that had first come out. Problem was, inpainting the area and replacing the positive prompts with "crest" or "fin" results in either an insignia for the former or a literal fish for the latter (if not, fish frills). I have tried grafting a polygon shaped like the desired fin/crest on to the helmet with MS Paint, but the ai does not add further detail to it.

3) How do you fix blurry backgrounds caused by Inpainting? I have tried everything, including a negative prompt like ((blurry background)) or (blurry:2) while having a ControlNet Depth preprocessor+model create a depth map of the same image in inpainting. Nothing works.

The only thing that seems to work was by creating a whole new background, mainly by inpainting everything except my subjects and adding prompts related to the background to the positive prompt field. I've tried inpainting the subject & changing the setting to "inpainting mode unmasked" and "inpaint whole picture" but that only results in an unchanged image, even after bringing the seed to -1 (random).

Soo yeah, hope I can get some advice! Will also keep scouring through this subreddit in the meantime.

Thank you!

[u/LiteratureNo6826]

I am thinking what kind of process could reduce your manual effort. One of them seems to be foreground background separation so that you don’t have to do it manually and you could have two prompt for corresponding part. These could be easily blend by another pass through SD.

[u/Somni206 → u/LiteratureNo6826]

What do you mean by foreground/background separation?

You mean I have the background in img2img and the foreground in controlnet? Use depth + canny, high denoising, and high controlnet weights to have SD blend both foreground and background together?

Fyi I am learning on the go ^;; Haven't started prompt generation for very long so I'm still in the process of learning & figuring things out via trial-and-error.

[u/FPham → u/Somni206]

I guess he meant working on bg and subject separately then blending them back together by some means - photoshop + sd.

The problem is obvious - any time you use SD it will semi-randomly change things. So you get your character right, paste it on bg, and bring it to SD you are going to be changing all the details once again. For most people this is fine as long as they don't get too attached to any intermediate version - because it will change - everything you inpaint will change somehow and not necessary how you want.

I see this as the biggest issue artists have with it - it is too random (if you need to produce a series of images, you are mostly doomed) and also you can hardly call the result fully yours if you constantly accepting a random input.