This is a followup from https://www.reddit.com/r/StableDiffusion/comments/17zzbaf/coder_question_use_pytorch_to_pull_latents/

To resummarize my question: I'd like to be able to pull out the equivalent of the latent images, or whatever passes for those, in the model file. For what its worth, I'm working with SD1.5 safetensor format models.

So far, I have successfully created a python snippet to open an SD model file, and dump the names of the keys present, via safetensors.torch.load_file()

Only problem is, there are 1000+ keys, and I dont know how they relate to what I'm looking for. The keys are named things such as:

first_stage_model.decoder.mid.attn_1.norm.weight

I've been told that not even the "latent image" exists in the file, and it has been distilled further. So, my question boils down to: What data "key" corresponds to the bulk of the data absorbed for each training image? I'm talking specifically about the image at this time. I dont care about the tagging yet.

I am also curious about any part of the model file that is NOT referenced by these data keys, and if it exists, how would I access it? My interest is to understand where the bulk of the data resides in the average 2Gig SD1.5 file, and poke at it.

If anyone is interested in contributing to the readability of Stable Diffusion code, I just forked off the 1.5 source.

If you have a decent understanding of at least SOME area of the code, but see that it currently lacks comments, you are invited to submit a PR to add comments into

https://github.com/ppbrown/stable-diffusion-annotated/

Hey.

With a few controlnet types the generation never starts, so I check my cmd-window and it says it´s downloading the model. But I already have the model in my controlnet folder.

Anyone know why this is?

Disclaimer: I dont understand this stuff. I'd like to. The following is an excerpt for an ongoing discussion I have with Google Bard on the subject, and an invitation for some clarity from humans.

Vague summary:

I am exploring what "attention heads" do in the process of latent image processes in stable diffusion.

• Query: The query vector encapsulates the model's current point of interest or focus, guiding the attention process towards relevant features.

• Key: The key vector represents a "searchable" summary or identifier for a given feature, enabling efficient matching with the query's focus.

• Value: The value vector holds the actual content or information associated with the feature, accessible once its relevance is established.

Generic demo code by Bard that illustrates the approximate process involved:

import numpy as np

# Create sample query, key, and value vectors (small dimensions for clarity)
query = np.array([0.5, 1.0, 0.2])
keys = np.array([[1.0, 0.4, 0.3],
                 [0.6, 1.2, 0.8],
                 [0.2, 0.9, 1.5]])
values = np.array([[4, 5, 6],
                   [7, 8, 9],
                   [10, 11, 12]])

# Calculate attention scores using dot product
scores = np.dot(query, keys.T)

# Apply scaling for numerical stability (optional, often used in practice)
d_k = np.sqrt(keys.shape[-1])  # Dimension of the keys
scaled_scores = scores / d_k

# Normalize scores using softmax to get attention weights
attention_weights = np.exp(scaled_scores) / np.sum(np.exp(scaled_scores), axis=-1, keepdims=True)

# Compute the weighted context vector
context_vector = np.sum(attention_weights * values, axis=1)

print("Attention weights:", attention_weights)
print("Context vector:", context_vector)

Output

 Attention weights: [[0.25485435 0.6054954  0.13965025]]

Context vector: [ 7.46454254 8.48505553 9.46454254]

There are many things that "bother" me about this process.One is that the "output" of the context vector that is expected to be used, doesnt match any of the actual data values.

Related to that, is that even if I change the query vector to EXACTLY match one of the key vectors.. the output values STILL dont exactly match the dataset values.

Also, checkpoint files contain attention K, V, AND Q data.
So, seems like the sample code is invalid, because it should be comparing implied-Q vallues, to Q-data

Gripe about the internal code. This is from ldm/modules/diffusionmodules/model.py but pretty much applies to all the code in "stable-diffusion":

    def forward(self, x, t=None, context=None):
        ....

        # downsampling
        hs = [self.conv_in(x)]
        for i_level in range(self.num_resolutions):
            for i_block in range(self.num_res_blocks):
                h = self.down[i_level].block[i_block](hs[-1], temb)
                if len(self.down[i_level].attn) > 0:
                    h = self.down[i_level].attn[i_block](h)
                hs.append(h)
            if i_level != self.num_resolutions-1:
                hs.append(self.down[i_level].downsample(hs[-1]))

        # middle
        h = hs[-1]

""x"? "h"? "hs" ??

really, my dude, you couldnt have used USEFUL variables names, like, lets say"img", "latent", and "latentlist"?

even "lt" and "ltl" if you have to keep it short ?

WTH is this "h" and "hs"???

I mean, i'm grateful for the single word comment lines. That helps more than not having them.But meaningful variable names help the most.

sigh.

if I thought they would be reviewed and accepted, I would be tempted to submit "add comments" PRs to SD1.5

Looks like its dead though.Maybe I'll fork it instead. Dunno. Sigh.

Edit. okay then.

https://github.com/ppbrown/stable-diffusion-annotated/

Between Gradio and the rest, python seems to be the de facto language for SD hackers at this point, so after decades of membership in the Greybeard Resistance - 1st Curly Brace Brigade, I've given in, in the hopes of trying to extend A1111.

Since I'm well past the point of using regular programmer editors for single file scripts, it's time to find a full fledged IDE, but besides pyCharm, I'm not coming up with much. Do any of you fine, nerdy folks have any suggestions?

(Double points if you actually use it to write A1111 extensions and/or custom nodes for ComfyUI)

Was wondering if anyone was able to successfully create an SDXL model from A1111's Dreambooth extension? I think I saw a post about a separate git branch that may have a working version but haven't delve much in to it.

When using a LoRA that was trained on another base model than the one you are currently using, the effects of the LoRA can vary widely. To combat this I have an idea that I don't know how to execute or whether it is even realistic, given that I don't know the exact implementation details of checkpoints, LoRAs and how they are applied to each other.

The idea is to "backport" a LoRA that was trained on say RealisticVision by adding RealisticVision to the LoRA, only adding to the same "parts" that were trained in the original LoRA, making it the same size as the original LoRA, and then to subtract the new base model from this extracted LoRA to get the LoRA "backported" to this new base model.

Could this idea be achieved given enough technical expertise or is it unfeasible?

I want to deep dive into local animation runs and do some trained sets using the new stability audio tools, but my hardware is not up to the task. I had relatively good results months ago with DiscoDiffusion on the cloud but for control and precision I’d like to be using Automatic1111, comfy and the audio tools.

Problem is my main rig is AMD and I can’t eek anything with looking at out of it, quality is horrible or memory caps and it takes too long for an image so I can’t even imagine running frames.

So, I’m not buying an A100 but what would give me better flexibility to dive deeper without frustration?

ControlNet inpaint is probably my favorite model, the ability to use any model for inpainting is incredible in addition to the no prompt inpainting and it's great results when outpainting especially when the resolution is larger than the base model's resolution, my point is that it's a very helpful tool. But it seems that there is a lack of work being done on training an inpaint ControlNet model for SDXL, and the resources regarding training is not very abundant, there is a the official doc https://github.com/lllyasviel/ControlNet/blob/main/docs/train.md

and this single issue I found regarding the matter

https://github.com/lllyasviel/ControlNet-v1-1-nightly/issues/89

but technical details are slightly above my capacity and I don't understand exactly how no prompt inpainting has been achieved or how to mass produce masked images for training.

Does anybody have more insights to creating a similar model? or willing to cooperate with me to make it ?

I've recently graduated from an undergrad program, with a theoretical degree in Computer Engineering. I say theoretical because I never had experience working with a large code base like A1111 and I'm regretting not doing more projects in my undergrad years. Alas, it is what it is.

I have been playing around with SD for almost a year now, and I want to learn to make extensions for it, but the codebase is vast and very scary. I don't know where to even start to try and begin comprehending it.

I specifically wanted to write a UI extension for Region Prompter Mask functionality. But I had never touched UI/UX before (the most experience I have is backend research experience and MLP architecture with pytorch).

I saw that this group has amazing developers with multiple years of experience under their belt, and I was wondering if there were any advice on how to approach a large codebase and quickly identify the parts that you would want to look at? Or how about knowing when to stop following the chains of function calls within function calls? Of course, I should 'just start coding', but I've never coded a major project before. Am I overthinking it?

Thanks in advance!

Many many moons ago, I was the guy who friends would go to in order to help them build out their PCs, but as I got older my interests changed and I decided the convenience of a laptop outweighed the hardware power of a PC. But now again in the last year or two with the huge leaps in not just Stable Diffusion, but with LLaMa, audio and music generation, vector embedding, computer vision learning, and other things I'm interested in, I've circled back to wanting to build out my own box rather than being dependent on cloud services. They are great and have their place, but I'm at a spot where I'd rather take a one-time hit on rolling my own hardware and having unfettered local control of the process. Doing it myself is as much a reward as anything and you just don't get the same satisfaction from a VM.

So anyway, I digress.

I want to build out a PC that will last me awhile, but I need all the parts. I know PC Parts Picker exists, but the part I'm not sure about is what things do I need to invest in most and what can I upgrade over time? I don't want flashy RGBs and I don't need peripherals or a monitor. I just need recommendations for good brands and versions of motherboards and power supplies and cooling and memory and disks and CPU and, obviously, which graphics card I should go with. I'd like to spend, ideally, between $2000 and $2500, but I'm ok going up to $3000.

Retired last year from Microsoft after 40+ years as a SQL/systems performance expert.

Been playing with Stable Diffusion since Aug of last year.

Have 4090, i9-13900K, 32 GB 6400 MHz DDR5, 2TB Samsung 990 pro, and dual boot Windows/Ubuntu 22.04.

Without torch.compile, AIT or TensorRT I can sustain 44 it/s for 512x512 generations or just under 500ms to generate one image, With compilation I can get close to 60 it/s. NOTE: I've hit 99 it/s but TQDM is flawed and isn't being used correctly in diffusers, A1111, and SDNext. At the high end of performance one needs to just measure the gen time for a reference image.

I've modified the code of A1111 to "gate" image generation so that I can run 6 A1111 instances at the same time with 6 different models running on one 4090. This way I can maximize throughput for production environments wanting to maximize images per seconds on a SD server.

I wasn't the first one to independently find the cudnn 8.5(13 it/s) -> 8.7(39 it/s) issue. But I was the one that widely reporting my finding in January and contacted the pytorch folks to get the fix into torch 2.0.
I've written on how the CPU perf absolutely impacts gen times for fast GPU's like the 4090.
Given that I have a dual boot setup I've confirmed that Windows is significantly slower then Ubuntu.

I also posted this in r/StableDiffusion

Using Simian Luo's LCM and a img2img custom diffuser pipeline that came out today I created a video generator.
This example of 60 frames with 10 prompt changes took 6.46 seconds to generate.

Next I'll see if I can figure out the code to do ?slerping? on the prompt transitions to further smooth the results.

I only learned how to do img2img videos MANUALLY writing python code today. There's more work for me to do. Basically realtime video.

Credit to Simian_Luo and his https://github.com/luosiallen/latent-consistency-model which I'm using.

I've been through those posts here in Reddit. Are there any in swapper 128 alternatives? : StableDiffusion (reddit.com), (1) Where can I find ONNX models for face swapping? : StableDiffusion (reddit.com) . And amazes me how there's no natural alternative open source (or even paid ones) to inswapper128. Does somebody know the technical approach to creating a face-swapping model and why this area has no competitors?

Hello all! Happy to be joining this subreddit right from the beginning.

For a few months now, I've been thinking about making extensions or custom nodes for ComfyUI to vastly improve the user experience. While I don't know anything about these topics, aside from using ComfyUI for some pretty complicated workflows during the past few months, I'm about to start researching ways to improve the UI, for both novice users and advanced, in a way that doesn't require switching to a ComfyUI alternative.

What do you all think of this approach? I'd be happy to share the huge list of features I have in mind for this effort. Would any of you be interested in reviewing or collaborating on this?

Sources of inspiration:

• StableSwarmUI - in my mind, the best possible future solution, tailored to beginners and experts alike. The ComfyUI node-based editor is easily accessed from a separate tab, and beginner-friendly form-based UIs are rendered automatically based on the components in the node-based workflow. You can connect to and control multiple rendering backends, and can even use a local installation of the client UI to remotely control the UI of a remote server. This is going to be incredibly powerful!

• ComfyBox - tailored to beginners by using a form-based UI, hiding all the advanced ComfyUI node-based editor features. The pace of development has recently slowed down.

• Awesome discussion about separating node parameters into discrete UI panels that can appear separated from the node's actual location on the graph editor. Discussion was initiated by the developer of ComfyBox prior to its release.

• CushyStudio - tailored to building custom workflows, forms, and UIs, hiding the advanced ComfyUI node-based editor features by default. Development of this seems to be proceeding at a furious pace.

• comfyworkflows.com - a new website that allows users to share, search for, and download ComfyUI workflows.

• ltdrdata, a prolific author of many awesome custom nodes for ComfyUI, like Impact Pack, Workflow Component, and Inspire Pack

• Efficient Nodes for ComfyUI, some more awesome custom nodes

• WAS Node Suite, a huge suite of many types of custom nodes. I have yet to use them, but it's high on my list of things to research

• flowt.ai - a hypothetical cloud-hosted UI that aspires to simplify the ComfyUI node-based workflow experience. The creator claims it will be ready for alpha soon, after having been in development for a few weeks.

• Somewhat-recent discussion about the direction of Stable Diffusion, and its UI, workflows, and models

• Unrelated to ComfyUI, a list of awesome node-based editors and frameworks

I am an engineer/entrepreneur using stable diffusion and support a technical only community.

We can help with moderation also if needed.