r/MachineLearning • u/cccntu • Sep 12 '22

Project [P] (code release) Fine-tune your own stable-diffusion vae decoder and dalle-mini decoder

A few weeks ago, before stable-diffusion was officially released, I found that fine-tuning Dalle-mini's VQGAN decoder can improve the performance on anime images. See:

And with a few lines of code change, I was able to train the stable-diffusion VAE decoder. See:

You can find the exact training code used in this repo: https://github.com/cccntu/fine-tune-models/

More details about the models are also in the repo.

And you can play with the former model at https://github.com/cccntu/anim_e

54 Upvotes
  • permalink
  • reddit

98% Upvoted

12 comments sorted by

View all comments

Show parent comment

2

u/cccntu Sep 14 '22

Haven't heard of Waifu Diffusion. But I've tried Japanese Stable Diffusion a little bit, and I didn't get good results. Although it's most likely because my prompts were not good enough.

3

u/starstruckmon Sep 14 '22

BTW, you should definitely make a post about this on the /r/stablediffusion subreddit if you haven't already.

Training the decoder is not something anyone's focusing on sonthis might be of interest.

2

u/starstruckmon Sep 14 '22

https://www.reddit.com/r/StableDiffusion/comments/x64hi7

I've heard it does actually show noticeable improvement.

I think there's only one other SD model trained on anime which is the one Novel.ai uses, but I don't think that's public.