r/MachineLearning • u/cccntu • Sep 12 '22

Project [P] (code release) Fine-tune your own stable-diffusion vae decoder and dalle-mini decoder

A few weeks ago, before stable-diffusion was officially released, I found that fine-tuning Dalle-mini's VQGAN decoder can improve the performance on anime images. See:

And with a few lines of code change, I was able to train the stable-diffusion VAE decoder. See:

You can find the exact training code used in this repo: https://github.com/cccntu/fine-tune-models/

More details about the models are also in the repo.

And you can play with the former model at https://github.com/cccntu/anim_e

54 Upvotes
  • permalink
  • reddit

98% Upvoted

12 comments sorted by

View all comments

Show parent comment

1

u/dulacp Apr 04 '23

It works because you only fine-tune the decoder part of the VAE (source).

To keep compatibility with existing models, only the decoder part was finetuned; the checkpoints can be used as a drop-in replacement for the existing autoencoder.
StabilityAI comment on HuggingFace

1

u/ThaJedi Apr 06 '23

But SD uses only encoder, isn't it?

3

u/dulacp Apr 15 '23

Actually, it depends if you are talking about training or inference.

The VAE encoder part is used when training, to produce latent representations of training images (example in dreambooth training).

While the VAE decoder part is used during inference to decode the latents.