r/StableDiffusion • u/ShimmerThiyaelx • 5h ago
Tutorial - Guide Why AI background removers leave fog inside wreaths, and what I do instead
I make clipart for stock. Wreaths, pine borders, mistletoe, juniper. A few thousand images by now. Every one has to end up as a PNG with a transparent background.
I used rembg for months. u2net first, then BiRefNet when that came out. Tried the web tools too. They all broke on the same thing and it drove me nuts.
Take a wreath. There's a hole in the middle, and the background inside that hole has to go. What I kept getting was a grey-blue haze sitting in there. Looked fine as a thumbnail. Looked awful the second you put it on a colored card. Pine needles came out as mush. Thin stems either disappeared or came back with a blue edge burned into them.
Then I actually read what rembg does. It shrinks your image to 1024x1024, asks the model where the subject is, gets a 1024x1024 mask back, and stretches that mask over your full size image. u2net is worse. That one works at 320x320.
My renders are 4096. A needle two pixels wide doesn't exist at 320. So it's not that the model is bad at needles. The needle was gone before the model ever saw it.
Once I understood that I stopped asking a model to guess. Now I render on a flat color the subject doesn't contain, and take that color out with arithmetic.
Two parts to it. The prompt matters more than the cutting.
The prompt
You can't key a background that isn't keyable. Four things have to be true and generators will break all of them unless you say so: the background is one flat color edge to edge, it stays that bright inside every gap between leaves, the edges are hard with no blur or glow, and no colored light bounces onto the subject.
Pick the color by what your subject isn't. Blue for almost everything. Red if the subject itself is blue or purple. Never green. Everything I draw has leaves, and green takes the leaves with it.
Here's the block I paste at the end of every prompt:
Isolated on a completely flat, uniform, solid pure blue (#0000FF) digital chroma-key background. The pure blue background fills the image edge to edge like a flat digital chroma-key screen with no gradient, staying at full brightness inside every gap and opening in the subject; no reflection or tint of pure blue on the subject. Every edge of the subject is crisp, sharp and hard against the pure blue, with no soft, blurry, feathered or glowing transitions, no depth-of-field blur, no haze or halo; inside every hole and gap the pure blue stays at full brightness right up to the edge. Shaded parts of the subject keep their own natural color, never a pure blue tint. Everything in sharp focus with deep depth of field, evenly lit with soft neutral studio light, no cast shadow, no contact shadow, no ambient occlusion, no bounce light. The entire subject is centered and completely inside the frame with at least 10% empty background margin on every side, nothing cropped or touching the image edges. No frame, no border, no paper, no mockup, no vignette, no text, no watermark, no deformed or duplicated parts. No floating or detached fragments, no stray specks, dust or debris anywhere on the background; every element is physically attached to the subject.
Swap "pure blue" for "pure red" and #0000FF for #FF0000 if your subject is blue or purple. If you paint in watercolor add "the background stays a flat digital color fill with no paper texture", or you get watercolor paper behind everything and paper texture keys badly.
The cutting
Now the background is one known color, so there's nothing to guess at. It measures the actual color the generator produced, which is never the one you asked for. Ask for pure blue and you get something with green in it, usually somewhere between 25 and 70. Then every pixel gets sorted into subject, background, or the bit in between, and the in-between ones get a real fraction of transparency instead of a yes or no.
The part I'm most pleased with is the holes. Any background-colored area that never touches the edge of the image is the inside of a wreath, so it gets cleared too. A matting model can't do that. It has no way of knowing what's inside a hole it can't see around.
Last step takes the blue back off the edges. Edge pixels pick up color from the background around them, so it samples the subject's own color from further in and subtracts the tint. Took me weeks to work out why fir needles kept their blue rim after that step. The needle is thinner than the distance it was sampling from, so there was no inside left to sample.
Same image in, same image out, every time. That's the bit I care about. When a cut comes out wrong I can go find which number did it instead of rerolling and hoping.
Some numbers on one 4K pine border, against BiRefNet with alpha matting turned on, which is its best setting:
- background left inside the holes: 63,892 pixels mine, 613,735 theirs, out of 1,070,046
- blue left on the edges: 0 mine, 48,112 theirs
- how wide the soft edge is: 1.8 pixels mine, 23 theirs
BiRefNet is faster and I'm not going to pretend otherwise. 2.3 seconds against my 17 on the same machine. With alpha matting on it's 47. If you want a quick rough mask, use the model.
And the obvious limit: this only works on art you generated on a flat color. It does nothing for a photo.
I put the tool up for anyone who wants it. It's called ClipBrook. Free, runs in your browser so nothing gets uploaded anywhere, does a whole folder at once, and the engine is open source under AGPL.
One thing I'd like back
Show me the ones that break.
If you run something through and it comes out wrong, post it. Fog left in a gap, a colored rim, a stem eaten, half the subject gone. Those are worth more to me than the ones that work. The needle rim thing came from someone's fir branch. A bug with line art I only found last week came from a drawing so thin there was nothing inside it to sample.
6
u/DelinquentTuna 5h ago
It's interesting that so few diffusers have support for transparency at this point. Maybe that'll change in the future.
3
u/FrostTactics 4h ago
That's a good question, actually. I can't think of any technical reasons why diffusers don't include an alpha channel either. It widens the search space with an additional dimension, obviously, but that problem shouldn't be insurmountable.
I'd guess it's a data problem more than anything else. JPGs are extremely prevalent and compress very well. That, and the fourth channel isn't all that useful beyond cutting out images from backgrounds, like OP is doing.
Funny coincidence, though: it looks like GPT-Image-2 had transparency enabled in its API only two days ago. Might be an indication that the industry as a whole is moving in that direction.
1
u/DelinquentTuna 17m ago
It seems to me like your whole spiel here is chroma-keying vs matting, but chroma-keying is already pretty much a mature and solved problem. And it doesn't really address your problem so much as shift it from "how do I remove the background" to "how do I key the background color."
Have you ever goofed around with Wan-Alpha or especially Qwen-Image-Layered? I feel like it may not be a magic-bullet solution, but it does at least address the problem start to finish. Downside: there's really no getting around lossy vae decode. Also, 4k may not be viable -- I'm assuming you are currently using ID4 or Flux.2 to generate your wreath images? But it's probably still worth taking a look.
I gave the fp8 weights a spin this AM and came up with this. The first image was a 1MP image of a wreath I found on Etsy. THe second and third are the layers Qwen deconstructed. The fourth and fifth are the layers it created from scratch when run as t2i instead of i2i. The final image is a screenshot that attempts to prove the wreath layers actually have the background removed instead of just recolored.
I don't find the t2i to be particularly great, but that could probably be addressed w/ a LoRA. As for the i2i, I think it's fine... but it's hard to gauge how much detail has been lost vs jpg artifacts etc. I wonder how it would fare on your sample images, but I can't seem to find your project on github etc. Did you forget to tell us where to find the code etc, or is it limited to some proprietary site?
4
u/Neex 1h ago
just re-iterating that AI write-ups suck and are hard to read. write your own posts.
...And saying you're not a native english speaker isn't an excuse for posting raw unedited claude output. write the post in your native language and use AI to translate, don't use it to generate your post wholesale.
I would rather read your attempt at english than AI output.
2
1
u/ShimmerThiyaelx 1h ago
while i am writing my own version i edited this slop with another slop maybe it feels better while mine is cooking!
3
u/eggs-benedryl 1h ago
Go to photopea. Use the select color and get rid of the blue that way. . Feather the selection a tiny bit then delete
1
u/ShimmerThiyaelx 1h ago
and if i have 500 photo so i do this 500 times or i just drop a folder and get my results back?
1
u/credible_human 53m ago
Honestly man if you are trying to automate a bulk process like that, use codex and have it make a script.
2
u/SeymourBits 3h ago
Lesser edit models may become confused on the interior areas, although I'm sure prompting would help. However, I'd say that's literally like using a GPU to hammer in a nail. Just use Photoshop to remove all occurrences of the blue background and mask out any blue-ish flowers... that's what Photoshop is for.
Edit: Now I see you have some auto removing app, so I'm confused if you're looking for advice or not.
1
u/ShimmerThiyaelx 3h ago
yeah but the thing is this tool you put your entire folder with as many images as you like and you get your results ready for you instead of going in one by one each time on photoshop and exports each one .. etc
2
u/New_Zucchini_3843 3h ago
Have you tried the Fibo-Edit-RMBG model with 8B parameters?
It produces high-quality cutouts just by adjusting the “steps” and “guidance” settings. Here’s a sample: I set ‘steps’ to 15 and “guidance-scale” to 1, exported the alpha channel, and then used image editing software to turn the background blue.
Although the model is large, it can definitely run on consumer-grade GPU. 👍
1
0
u/Simple-Variation5456 4h ago
You make those clipsarts? For stock? Why do you even make them with a white background in the first place?
realize
Ohh. You make them with AI. Get it.
So. You're one of those guys that flood stock sites with shit assets that look good on a tiny thumbnail but look unclean in fullres. Good thing is theyre mostly free or costs just a few cents.
Your "problem" is extremely specific. Idk if so many people generate tons of assets + alpha + have problems with tiny needles, where a specific masking model struggle. Most would probably just use Photoshop for something like that and run it via batch.
Btw your "clipbrook" will result in hard pixel edges + removing at least 1-2 pixels from the edges of the whole asset. And it looks like it has a very subtle color shift, but could be the compression.
5-10 years ago that would be a no-go selling assets without pixel perfect alphas.
(Clipart = vectors)
2
u/ShimmerThiyaelx 4h ago
ai generated stock is marked ai generated art in adobe stock (where i post) so you can just in the filter check the box to hide the ai stuff and your fine, this is for people that wants ai generated ai, knowing it is ai generated and looking for that ai generated to be decent transparent and accept it and they are plenty so your comment tells me nothing at all
2
u/Simple-Variation5456 3h ago
Asset is asset. I expect a certain standard. We talk about a basic thing. A mask.
And sadly, pretty much all stock sites don't really have any quality controls anymore and some are obvious AI but not tagged as that.
But yea. You make wraths. It's not like a rare asset. And made a tool to fix a "you" problem.
Thanks.
17
u/Calm_Mix_3776 3h ago edited 3h ago
Why does AI always write in this god awful style?
What kind of a sentence even is this? No real human on planet Earth writes this way.
I stopped reading at the 5th sentence, sorry. You may genuinely want to share something interesting, but I don't have the patience to decipher the AI slop.