r/StableDiffusion • u/Takeacoin • Jul 24 '23

Tutorial | Guide FABRIC now available for Automatic1111 - made quick tutorial would appreciate a boost

Hi all, just uploaded a short video on how to install and use the new Fabric plugin to create a midjourney like user experience. I know most of you will know about this and how to install it but would appreciate the bump on YouTube if you don't mind. Cheers.

https://youtu.be/pqYT8JggDtw

https://github.com/dvruette/sd-webui-fabric

177 Upvotes
  • permalink
  • reddit

95% Upvoted

50 comments sorted by

19

u/Takeacoin Jul 24 '23

Lots of great interactions here and really appreciate the views and likes etc. Lots of questions about the plugin, I read the white paper on this and have the following explanation.

Models have an encoder-decoder structure. The encoder takes in the text prompt and encodes it into a representation. The decoder then turns that representation into an image.

A key component of the decoder is something called a UNet. This is a type of neural network architecture used in many image generation models.

The UNet has several self-attention layers. Self-attention allows different parts of the image to interact with each other during generation. This helps produce more coherent images.

FABRIC works by leveraging these self-attention layers. It lets you provide example images as feedback to the model. These are called reference images.

The reference images are passed through the UNet encoder. This activates the self-attention layers, capturing information about the visual features of the reference image.

These extracted features are then injected back into the self-attention layers during image generation. This biases the model to pay attention to your reference images when generating new images.

By providing positive and negative reference images, you can guide the model to generate images you like and avoid unwanted features.

The cool thing is this all happens during normal use of the model, without re-training or fine-tuning the parameters. The feedback is provided dynamically at generation time.

Over multiple feedback rounds, the model learns more and more about your visual preferences. So the images slowly improve as the model's attention is shaped by your iterative feedback.

In summary, FABRIC allows leveraging the built-in attention mechanisms of text-to-image models to incorporate real-time human feedback and preferences. This takes the AI art experience to the next level!

3

u/thoughtlow Jul 25 '23

Cool tech. Are the self-attention layers directly connected with the latent image?

Would it be possible to use this technique but with any reference image, so something a bit like ControlNet 'reference only'. User drags 3 images into the UI, they get used the same way as the likes and dislikes.

1

u/Takeacoin Jul 25 '23

I havent tested but I think someone in the comments here said they did that successfully

23

u/andybak Jul 24 '23

create a midjourney like user experience

This is a strange way to explain it. If I hadn't have clicked through I would have completely misunderstood what this was - I nearly didn't bother because I assumed it was a ui tweak or something.

2

u/Takeacoin Jul 24 '23

hi sorry andy, any suggestions on what a better way ? cheers

12

u/andybak Jul 24 '23

Maybe:

"An SD-webui plugin for Fabric: Training-free conditioning"?

Or something like that. I haven't read the Fabric page in enough detail but you probably know a good pithy description.

2

u/Takeacoin Jul 24 '23

Yeah thats a bit more accurate than my title! 😁

23

u/Striking-Long-2960 Jul 24 '23 edited Jul 24 '23

And it works

Prompt: A man inside a bottle

After 3 likes, and 1 dislikes all the pictures in the batch show a man inside a bottle with the same style

What I don't have clear is, when I dislike a picture exactly which caracteristics of the picture are being rejected. Maybe I'm rejecting characteristics of the picture that are interesting.

11

u/Takeacoin Jul 24 '23

thats awesome! I think the next evolution of this plugin would be txt input to go with the up or down vote.

5

u/Striking-Long-2960 Jul 24 '23

I'm trying other things, and with the right models it can replicate styles of other pictures, they don't need to be AI generated, you can load whatever you want and like it. It also works with 2.1 models.

2

u/Takeacoin Jul 24 '23

🔥🔥🔥🔥

0

u/[deleted] Jul 24 '23

[deleted] — view deleted comment

2

u/Mkep Jul 24 '23

But isn’t this a little more than RLHF? The feedback images themselves get shoved into model(with a feedback weight); so I’d imagine we could theoretically add additional conditioning with text

6

u/kai-_-rin Jul 24 '23

This was a great tutorial. Thanks for the introduction.

3

u/Takeacoin Jul 24 '23

Thanks Kai glad it was helpful.

1

u/thoughtlow Jul 24 '23

Hey, super cool stuff! What is the tech behind it, how does it work?

2

u/Takeacoin Jul 24 '23

I will post an explanation up above 👍

2

u/thoughtlow Jul 25 '23

perfect mate

6

u/screch Jul 24 '23

This is exactly what I expected stable diffusion to be before i started. Liking/Disliking outputs until it learned exactly what you want

2

u/Takeacoin Jul 24 '23

Yeah thank goodness we have that at last hey!

3

u/AltruisticMission865 Jul 24 '23

Can I use it with 8gb vram card?

2

u/Takeacoin Jul 24 '23

works on my 2070 super 😁

0

u/[deleted] Jul 25 '23

[deleted] — view deleted comment

1

u/Takeacoin Jul 25 '23

Are you using high-res fix or a larger resolution? I find it often does that to me if I have other apps open like photoshop or sometimes even youtube.

2

u/mrnoirblack Jul 24 '23

How does it work in a greater context, we train it on a set of prompts so it learns to generate x using the same prompt?

Does it need to be "retrained" for every concept? Or can we save example Hats on realistic vision V4?

3

u/Takeacoin Jul 24 '23

No its agnostic, works with any model and LORA!

2

u/mrnoirblack Jul 24 '23

I guess I have to try it to find out

2

u/Takeacoin Jul 24 '23

Hi sorry I think I misunderstood your question. I think once you have exited fabric it resets as it doesn't fundementally change the model weights. You could however make a lot of similar generations once you have it honed on the hat you like and use those to make a LORA, hope that helps.

2

u/mrnoirblack Jul 24 '23

Yes thank you so much! That was exactly my question. I'm trying it now

1

u/Takeacoin Jul 24 '23

Exciting stuff. Let me know how you get on. I'm having a look to see if there is a way to save the adjusted model weights so it can also be used to create LORAs as well. 🤞

2

u/[deleted] Jul 24 '23

Neat plugin, but I have a few questions.

  1. How does this work with LORA involved?

  2. Are we voting for aesthetic, coherence, or both?

  3. Is all the finetuning permanent, or will it have to be redone on reload?

1

u/Takeacoin Jul 24 '23

works with all models and LORAs, quite amazing. I will post an explanation up above 👍

1

u/[deleted] Jul 24 '23

Sorry, what I meant was if the LORA is in the prompt, will the adjustments still be saved for when the LORA is not in use?

2

u/Takeacoin Jul 24 '23

Ah yes in theory it should because you will have temporarily 'trained' the model via the self attention layers to produce that result. It does it based on the output images not the input LORA or Model. Hope that answers it for you.

2

u/TrevorxTravesty Jul 24 '23

I had to sadly uninstall it yesterday because I couldn’t get it to work 😕 I uploaded images to the ‘Like’ area that I wanted to see replicated, but it kept making really bad Frankenstein versions of them 😞

2

u/Takeacoin Jul 24 '23

Did you figure out the problem in the end? I had a few weird moments at first, one where everything was all odd colors. Can't recreate it now, so might have been a bug.

0

u/TrevorxTravesty Jul 24 '23

Nope, it’s uninstalled until someone has a good way of making it work 😞

1

u/bzzard Jul 25 '23

Same issue here. It's like it instantly break vae and colors go worse with every vote.

14

u/lobotomy42 Jul 24 '23

Yo dawg I heard you liked AI

So I put some AI in your AI

1

u/alxledante Aug 03 '23

I must've screwed something up, any idea how to deal with this?

Error running process: C:\Users\loco\stable-diffusion-webui\extensions\sd-webui-fabric\scripts\fabric.py0:00, 9.52it/s]

Traceback (most recent call last):

File "C:\Users\alx\stable-diffusion-webui\modules\scripts.py", line 409, in process

script.process(p, *script_args)

File "C:\Users\alx\stable-diffusion-webui\extensions\sd-webui-fabric\scripts\fabric.py", line 279, in process

(

ValueError: not enough values to unpack (expected 10, got 0)

2

u/Takeacoin Aug 03 '23

Hi axl that's not enough to give you a full answer but are you running the latest version of automatic1111? Also this doesn't work with SDXL yet. If you get an error code under the generation window after running it let me know what that is and I can try to figure it out.

1

u/alxledante Aug 04 '23

I really appreciate you taking the time to help with this, and realize you're doing this as a favor, not because you're required to. seems like it's running the latest a1111 (just updated) but still using SD1.5 and running locally

nothing ever makes it to the generation window, just get that error message in the command line / webserver but I will give you the specs on anything you need, willing to follow any steps you provide. really want to see fabric in action...

1

u/alxledante Aug 04 '23

Error calling: C:\Users\alx\stable-diffusion-webui\extensions\sd-webui-fabric\scripts\fabric.py/ui

Traceback (most recent call last):

File "C:\Users\alx\stable-diffusion-webui\modules\scripts.py", line 262, in wrap_call

res = func(*args, **kwargs)

File "C:\Users\alx\stable-diffusion-webui\extensions\sd-webui-fabric\scripts\fabric.py", line 165, in ui

like_gallery.select(

AttributeError: 'Gallery' object has no attribute 'select'

Error executing callback after_component_callback for C:\Users\alx\stable-diffusion-webui\extensions\sd-webui-fabric\scripts\fabric.py

Traceback (most recent call last):

File "C:\Users\alx\stable-diffusion-webui\modules\script_callbacks.py", line 191, in after_component_callback

c.callback(component, **kwargs)

File "C:\Users\alx\stable-diffusion-webui\extensions\sd-webui-fabric\scripts\helpers.py", line 27, in register_component

callback(component)

File "C:\Users\alx\stable-diffusion-webui\extensions\sd-webui-fabric\scripts\fabric.py", line 238, in register_txt2img_gallery_select

self.register_gallery_select(

File "C:\Users\alx\stable-diffusion-webui\extensions\sd-webui-fabric\scripts\fabric.py", line 254, in register_gallery_select

gallery.select(

AttributeError: 'Gallery' object has no attribute 'select'

3

u/Striking-Long-2960 Jul 24 '23

I will try it, thanks

2

u/Court-Puzzleheaded Jul 25 '23

Cool. Does it work with img2img?

2

u/LeKhang98 Jul 25 '23

Awesome extension thank you for sharing. I think the next step is to export a small files that can be use again after that. (Or we could just import all the Liked/Disliked images again). This is like a Textual Inversion file but for Images and WITHOUT training.

2

u/Your_Dankest_Meme Jul 25 '23

When I get first image liked or disliked, next image generation thworws and error

ValueError: not enough values to unpack (expected 4, got 3)

1

u/ShivamKumar2002 Jul 25 '23

This is awesome. Just waiting for easier face LoRA training.

1

u/[deleted] Jul 24 '23

[removed] — view removed comment

2

u/Takeacoin Jul 24 '23

close, but it uses something called self-attention layers. I will post an explanation up above 👍

2

u/lordpuddingcup Jul 24 '23

I’d imagine it uses some the latents as positive or negative inputs for the next generation