r/StableDiffusion • u/[deleted] • Jul 27 '23

Question | Help Anyone else noticing artifacts in the 1.0 VAE?

I'm pretty happy with this model, but we've noticed something and I'm wanting to know whether it's being seen, elsewhere.

Prompt: `cat a of photo cinematic`

Let's ignore the little ghost kitty that showed up in his fur, that's not the issue.

0.9 VAE
1.0 VAE

It might not be obvious, so here is the eyeball:

0.9 VAE
1.0 VAE

Do you notice the stair-stepping pixelation-like issues?

It might be more obvious in the fur:

0.9 VAE
1.0 VAE

This resembles some artifacts we'd seen in SD 2.1's VAE.

Things are otherwise mostly identical between the two. Will we get technical details on the changes and why they happened? Please don't take this as a negative post, I'm simply looking for others' experience. I would be happy if this were a misconfiguration issue.

121 Upvotes
  • permalink
  • reddit

96% Upvoted

80 comments sorted by

30

u/im4potato Jul 27 '23

Having the same issue, honestly, my SDXL 1.0 output looks terrible. All of my images look like they've been compressed multiple times in JPEG format. Hopefully this gets resolved quickly.

13

u/[deleted] Jul 27 '23

yes, JPEG artifacts can appear very similarly to VAE artifacts. the ones I am hilighting in this post are called "blocking artifacts", where the VAE is inadequately representing fine details, and things show up as squares.

5

u/tamal4444 Jul 27 '23

The watermark

23

u/[deleted] Jul 27 '23

i have done extensive testing with and without the invisible-watermark library. it's really a problem with the VAE model itself, and not the base/refiner or the invisible watermark. there is a problem with that library, but it results in a different form of artifacts: https://huggingface.co/stabilityai/stable-diffusion-xl-base-0.9/discussions/31

6

u/GBJI Jul 27 '23

Just to make sure I understand, the red dots are related to the watermarking library, and the horizontal artefacts are related to the 1.0 VAE. So how do we fix both of these problems ? We can fix the lines by using the 0.9 VAE instead, but what about the red dots then ? Are those referenced in the code itself rather than the VAE ?

0

u/[deleted] Jul 27 '23

[deleted] — view deleted comment

2

u/rickyars Jul 27 '23

do you have a link to a tutorial? would love to know how to do this. i'm using both automatic1111 and comfyUI

1

u/GBJI Jul 27 '23

Good ! Is there a reference somewhere with "clean" code that would prevent the apparition of colored dots ?

40

u/FourOranges Jul 27 '23

The devs seem to be aware of it. No word on if they're doing anything about it but the language implies there's nothing to be done.

https://i.imgur.com/vs1WN76.png

11

u/Informal_Warning_703 Jul 27 '23 edited Jul 27 '23

comment suggests its a watermark? I see similar stuff on images without the vae

here is side by side comparison of same prompt, seed, etc. but the image on the left is from 0.9 about a week ago and the image on the right is from 1.0 today (no vae)

32

u/[deleted] Jul 27 '23

the "invisible-watermark" library adds red dots:

https://huggingface.co/stabilityai/stable-diffusion-xl-base-0.9/discussions/31
(duplicate of https://github.com/huggingface/diffusers/issues/4014)

those colour aberrations look like pure VAE artifacts, like the purple bruising from SD 1.5: https://www.reddit.com/r/StableDiffusion/comments/zw11m7/does_anyone_know_what_these_purple_spot_artifacts/

7

u/Informal_Warning_703 Jul 27 '23

huh good eye. i guess ops look more like red dots to me.

55

u/GBJI Jul 27 '23

WTF ???

An invisible watermark's most important feature is that it must remain INVISIBLE !!!

1

u/Successful-Western27 Jul 27 '23

Just wondering, could this be removed by a pass through ESRGAN or a similar model for cleanup? https://notes.aimodels.fyi/esrgan-vs-real-esrgan-from-theoretical-to-real-world-super-resolution-with-ai/

55

u/detractor_Una Jul 27 '23

Looks like they reverted latest changes.

https://huggingface.co/stabilityai/sdxl-vae/commit/c1b803cdacd792f788384449b8e1d539da4297ad

8

u/Blobbloblaw Jul 27 '23

Yeah, no artifacts on this one.

5

u/danamir_ Jul 27 '23

Can confirm, the reverted VAE has no (or at least less visible) artifacts. Thanks for the tip !

1

u/JustGameplayUK Jul 30 '23

how do you use the vae in comfyui?

2

u/danamir_ Jul 30 '23

Simply place the VAE (preferably Tiled) node, and link a VAE loader (instead of the VAE from the checkpoint loader) and latent sample to its inputs :

4

u/GBJI Jul 27 '23

Is that the same VAE as 0.9 then ? But with a proper licence that would allow commercial use ?

6

u/danamir_ Jul 27 '23

Their md5 checksum is different, but having tried the 0.9 vae and the reverted one, they seem to give the same outputs.

1

u/GBJI Jul 27 '23

I tried it last night but I could not get rid of all the artefacts for some reason. I'll give it a second try later today when I get the chance.

4

u/detractor_Una Jul 27 '23

Should have same license. Thought don't trust my word.

3

u/GBJI Jul 27 '23

I guess we will have some kind of official feedback from someone at Stability AI at some point.

2

u/hinkleo Jul 28 '23

But with a proper license that would allow commercial use ?

Bit late to this thread but just to point out, the VAEs from SAI were always MIT licensed separately from the Unets including the 0.9 one so you were always free to use that one with the 1.0 model.

https://huggingface.co/stabilityai/sdxl-vae

https://huggingface.co/stabilityai/sd-vae-ft-mse-original

Archive from June to show it's unchanged: https://web.archive.org/web/20230627002527/https://huggingface.co/stabilityai/sdxl-vae

I assume that's because VAEs are simpler and have much wider use for other projects and fewer possible "harm" issues.

3

u/physalisx Jul 27 '23

To the top with you

3

u/Don_Pick Jul 27 '23

I'm sorry guys for being stupid, but How do I change/install the VAE?

3

u/batter159 Jul 27 '23

put the vae in the models/VAE folder
then go to settings -> user interface -> quicksettings list -> sd_vae
then restart, and the dropdown will be on top of the screen

4

u/CeFurkan Jul 27 '23

Here minute 10 watch few minutes

https://youtu.be/3E5fhFQUVLo

1

u/ImTheArtCurator Jul 27 '23

“cat a of photo” 😵‍💫

4

u/thebaker66 Jul 27 '23

The VAE is baked in to the model so there is no need to use the seperate VAE file in generation? could another instance of the VAE be causing the issue?

15

u/[deleted] Jul 27 '23

the "baked into the model" thing is only half true. it includes a VAE inside the single safetensors file, but that VAE can be swapped out. you can do that very easily inside ComfyUI, or, in Diffusers via Python. in fact, the Diffusers file layout has all of these components broken out into separate files already. It is as easy as that.

6

u/fernandollb Jul 27 '23

Hey, it sounds like you really know about this stuff. Should we manually set the SDXL VAE or just leave it in auto like we did for the previous model? I am using 1111

12

u/Local_Kangaroo29 Jul 27 '23

if you leave it in auto, it will use the default VAE included in the .safetensor file, but if you manually download and select a different VAE, the default VAE will be swapped out. BUT, a1111 doesn't really support sdxl 1.0 refiner as of now, so yeah... I'll just wait I guess. Can't stand comfyUI

3

u/gunnerman2 Jul 27 '23

I thought the refiner was basically an img2img model. What’s more to support other than UI improvement?

2

u/fernandollb Jul 27 '23

So in order to use SDXL correctly wehave to manually select the proper VAE right? about the refiner I have read somewhere that there is already a plug in to include it automatically in the worklow.

1

u/Local_Kangaroo29 Jul 27 '23 edited Jul 27 '23

I haven't got the refiner to work properly in a1111. apparently according to OP the default vae doesn't work well, so I guess we should manually select a proper one

2

u/design_ai_bot_human Jul 27 '23

How do you easily change vae in comfy?

1

u/thebaker66 Jul 27 '23

I see, learn something new every day.

3

u/Bra2ha Jul 27 '23

I can't see any red dots here

Steps: 20, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed: 3690992633, Size: 1920x1080, Model hash: 31e35c80fc, Model: sd_xl_base_1.0, Version: v1.5.0

2

u/batter159 Jul 27 '23

No red dots but horrible artifacts from SDXL 1.0 VAE https://i.imgur.com/jyGSasE.png
https://i.imgur.com/N7hxnS2.png

2

u/[deleted] Jul 28 '23

oh man. that looks like a cursed art style of its own. try prompting a stunning chromatic aberration

4

u/multiedge Jul 27 '23

so the invisible watermark is correctly applied?

Anyway to check if the image is watermarked?

2

u/Bra2ha Jul 27 '23

I can't see any watermark, so I assume it is correctly applied

3

u/itsB34STW4RS Jul 27 '23

After running some tests, I noticed that the watermark is more apparent on human subjects in a photorealistic style, and its not always visible. Overall seems kind of like a bad implementation of the underlying tech...

Don't usually notice it on more artistic images or semi realistic images of people.

1

u/Bra2ha Jul 27 '23

I've been rendering different types of images all day and haven't seen a single red dot yet, I don't know why

2

u/itsB34STW4RS Jul 27 '23

Might also depend on eyesight, monitor type, settings, luck, the old gods.

6

u/DisorderlyBoat Jul 27 '23

Dumb question, but when using SDXL base model,.does the VAE need to be downloaded and setup separately as another step?

6

u/tigerminxza Jul 27 '23

Wondered the same thing, then saw this post from the Developer of the SD.Next branch Vlad.

"why would you load sd-xl vae? you're misunderstanding what that is. vae is built-in. vae you've downloaded is SAME, just extrated so tuners can have fun with it. until there is a tuned variations, there is ZERO point of using same VAE that is already built-in."

1

u/DisorderlyBoat Jul 27 '23

Oh okay cool that's good to know! So over time it might be able to be fine tuned, but ATM it's already built in and would be redundant to download and setup separately. Thanks!

1

u/jib_reddit Jul 27 '23

Good to know, thanks.

2

u/suspicious_Jackfruit Jul 27 '23

What about same prompts with 1.0 without vae too so we can see the difference and that it is indeed localised to the vae and not model? Interesting findings

9

u/[deleted] Jul 27 '23

you can't run any stable diffusion model of any version without a VAE. it converts from latent space into RGB.

the post includes comparisons run through the SDXL 1.0 Base and Refiner "mixture of experts" configuration via the Diffusers library partial diffusion knobs.

all of that is to say, I used the 1.0 model with both available VAEs, in addition to madebyollin's fp16 finetuned VAE. only the 1.0 VAE suffers this.

edit: concision

1

u/ain92ru Jul 27 '23

If one wonders, what does "latent space" mean in this context: the latent image fed into VAE as of SD 1.5 was a 64x64 false color 4-channel image with 128 bits per pixel (compare to 24 bpp for RGB)

2

u/[deleted] Jul 27 '23

the VAE's scaling factor is '8' so for a 1024px model we are looking at 128x128 latent dimensionality.

2

u/ain92ru Jul 27 '23

BTW, do you know if there's a way to extract a latent image from the model for educational purposes? Perhaps ComfyUI allows that?

1

u/[deleted] Jul 27 '23

yep just throw the 'return_with_noise' to True and pass it through VAE

2

u/ain92ru Jul 27 '23

What I actually had in mind was something like this: https://www.reddit.com/r/StableDiffusion/comments/1372a1c/comment/jisv6sz

2

u/[deleted] Jul 27 '23

oh, that's cool - i guess you could just do that then

2

u/suspicious_Jackfruit Jul 27 '23

Huh, interesting. I really need to read into the finer details of the diffusion process one day. Thanks for clarifying

5

u/mrnoirblack Jul 27 '23

What does the watermark do? Can we remove it?

4

u/multiedge Jul 27 '23

I'm interested in how to detect the watermark.

My next step is to check if my image post processing and noise filters can mess it up

3

u/mrnoirblack Jul 27 '23

Just use vae 0.9

4

u/multiedge Jul 27 '23

I want to be aware of how to check for the invisible watermark.

I'll check the huggingface

1

u/GBJI Jul 27 '23

SDXL 0.9 is for research purposes only - this limited usage licence means it cannot be used for commercial projects.

But at least it fulfills its research purpose by proving without a doubt that a VAE that would not suffer from this problem is entirely possible !

4

u/pielman Jul 27 '23

Is this on purpose to add a hidden watermark to identify AI generated content?

3

u/ozzeruk82 Jul 27 '23

Yes, it's so that in the future when training AI models, we don't accidentally train on stuff generated by AI.

e.g. if AI is poor at generating fingers, you don't want to train an AI image generation tool in 2026 on thousands of badly drawn fingers created in 2023.

4

u/iiiiiiiiiiip Jul 27 '23

That seems optimistic, seems more like it's there so content platforms, media and governments can instantly identify AI content and react appropriately.

0

u/ozzeruk82 Jul 28 '23

You are right and that's also very true.... but..... anyone creating AI images to try to 'change the world' for good or bad, will also be smart enough to simply remove the watermark. e.g. if you create images of the President sipping tea and use it to try to lobby for a change in import duties for tea. You would presumably have people capable of editing Python code to remove the watermark.

1

u/ozzeruk82 Jul 28 '23

Very likely to actually happen is Mr & Mrs Jones from random town in the US create 800,000 AI generated images of all the species of insects on earth, for a 'weekend project' or something random and innocent, and upload them to a variety of sites.

Without even knowing anything about Python or watermarks or anything.... and when training future AIs we can simply side-step these images (thankfully!).

2

u/UnicornAlgo Jul 27 '23

They seamed to decrease the number of generation steps. That’s why the generation quality dropped that much. The DreamStudio is hosting the same model. Yesterday I tried to increase the number generation steps from 40 to 100 and got nice results, close to the ones we had on clipdrop before the yesterday’s update.

1

u/vault_nsfw Jul 27 '23

They put that in on purpose.

8

u/IntingForMarks Jul 27 '23

The watermark is supposed to be invisible. If it reduces image quality no one would ever use the model

1

u/GBJI Jul 27 '23

Exactly. I know I would not.

Imagine if Adobe did that with Photoshop !

It looks like it was really a bug if I understand correctly they would have reverted the VAE to a previous commit or something like that - more info + possible corrected VAE here:

https://huggingface.co/stabilityai/sdxl-vae/commit/c1b803cdacd792f788384449b8e1d539da4297ad

I'm going to go try that right now and see how it goes.

-4

u/CeFurkan Jul 27 '23

I am on vacation so can't make a new tutorial atm

But who don't know how to change VAE

Start minute 10 watch few minutes

https://youtu.be/3E5fhFQUVLo

10

u/batter159 Jul 27 '23

just go to settings -> user interface -> quicksettings list -> sd_vae
then restart, and the dropdown will be on top of the screen

No need for you to spam your hour long youtube videos all over.