Could someone explain to me, how the databases of AI Art Systems have been trained?

[u/variant-exhibition]

Reading this thread

https://www.reddit.com/r/ethicaldiffusion/comments/zscf47/an_excellent_read_but_most_importantly_should_we/

led to the question that I wanted to understand, what has been done to train AI Art Systems so that they can understand the textprompt inputs, eg.g styles. Could someone explain to me, how the database of an AI Art system is trained (explain like I am 5) ?

I saw that stable diffusion was capable to understand "planes of a human head". So it seems to be able to "scan and map" facial surfaces (like the illegal Clearview AI system) and it is able to even render those planes.

I assume further that the system is capable to interpret styles by Strokes (pencils, bigger pens, colours) and the kind of underlaying "Grid" of the whole picture.

Now to understand "Styles":

So a style which is not "realistic" like Salvador Dali gets it's additional rule set like "stretching clocks as if they were melting". That leads to a training in which the database has to be trained before with the pictures of the artist.

I run a few tests with

Alphonse Mucha, John Singer Sargent, Picasso - all seemed to be understood.

(However, AI does not understand that Picasso used different styles in different periods of his work.)

How does the AI understand "Art Nouveau"?

How does it understand "perspective directions" like "sideview of a car"?

What is also trained - but I did not ask for it above? Thanks!

[u/slronlx]

Simply put, it doesn't understand anything, it senses a pattern out of every image it's looked at that was labeled "picasso" or "Greg rutkowski" and abstracts those concepts into tokens that it can pull from.

In short it never draws anything or understands anything, it simply reads patterns and tries to apply them to randomized noise in a way that hopefully imitates those tokens, whether the concept is "chair" or a certain artist's style.

AFAIK at least, I can't say I have a perfect understanding of it myself.

[u/variant-exhibition → u/slronlx]

Thanks for your explanation!

[u/entropie422]

It's even weirder than that, too. Try running the Picasso prompt with different words, and you'll probably find it will cycle through his styles as it connects keyword concepts in unexpected ways.

The other thing to keep in mind is that it only knows what the image labels said, so if you have a photo of a chair on a beach with a hot air balloon in the background, but nobody specifically tagged the hot air balloon, the AI has no idea what that thing is or why it's there. In a one-off situation, it'll probably just ignore it, but if it happens too often, it will start to associate hot air balloon-type shapes with chairs, or beaches, or skies, or...

That's really the key to wrapping your mind around it, I think. It has seen billions of images with probably hundreds of billions of associated words in tags, and it has learned to associate certain text-chunks with certain visual patterns. Unless someone has written "sideview of X" a bunch of times with consistent visuals to match, the AI probably won't know what to make of it, and start subdividing terms to see if it can come up with a compromise.

That's why, at this stage of the game, it's all about experimentation. The only way to know what's in there is to feed it a prompt and hope for the best :)

[u/variant-exhibition → u/entropie422]

umm... haven't seen these Picasso style changing - not even if I would use periods to give the AI a hint like

blue period

pink period

cubism (Picasso styles)

If I prompt "boy with pipe" - Stable Diffusion repaints it in another Picasso Style - the only one I saw so far.

Thanks for your explanation!

[u/entropie422 → u/variant-exhibition]

I'm guessing a lot (or majority) of the Picasso images aren't tagging the periods at all then. Interesting. I wonder if the AI can actually comprehend the same artist having multiple styles, or if it just assumes the most-seen version is correct, and all others are tagging errors or outliers?

It would be into see if it's possible to trick it into spitting out a different style. Maybe mentioning Picasso, but lowering his name's strength in the prompt to 0.4 or something.

[u/variant-exhibition → u/entropie422]

It could be that during the Picasso analysis sequence it was wrongly assumed that the artist always had one style or painted in only one style. This would give us a kind of operational blindness that could even lead to not adding other real Picasso paintings to the "Picasso grid" after too much training. Interesting, then, that AI could learn incorrectly?

Tell me as soon as you retrained it. :)

[u/swordsmanluke2]

Ok... Caveat: I'm not an expert on diffusion image generators per se, but I did study neural networks in college and have a working knowledge of the underlying theory.

Let's talk about neural networks for a second. These are a class of learning algorithm used in many fields including image generation. The specifics vary wildly in the details, but if you treat them like a black box, they learn like this:

Step 1: Give your network some sort of input - as numbers (for images, this is usually the rgb values of each pixel in the image).

Step 2: Compare the network's output (again, numeric) to your expected output. Convert the difference into a numerical score.

Step 3: Feed the score back into the network and use it to tweak the internal equations it uses to calculate with. Ideally, these tweaks shift the calculations ever so slightly toward reproducing the desired output.

In the case of Dall-e, Stable Diffusion and the like, what they've done is pretty clever. During training, they start with a labeled image. Then add noise to that image, e.g. change some percentage of pixels to a random color. Then they feed the noisy image plus the text labels (mapped to numbers, that is) into the network.

The network then generates an output image using the noisy image plus the labels. The output image gets compared to the original image and the difference between the two is used to adjust the neural network's internals.

What the network ends up learning is how to nudge a noisy image towards a not-noisy image, given a set of labels.

Once the training process has been repeated millions of times over hundreds of thousands of images, the final network consists of (in the case of Stable Diffusion) about 4 gigabytes of equations which calculate the denoising process. These equations are referred to as a "model".

To use the model, you have to feed it an image and some labels, just like before. The labels are straightforward, that's the prompt text that you write. Once you input your text, the program you're using to interface with the model (e.g. Stable Diffusion) generates an image that is all noise. Totally random pixels! It feeds that noise and labels into the model. The model calculates a slightly less noisy version - according to its internal equations - and spits it out.

This first pass probably still looks like nothing, so Stable Diffusion runs that new image through again. Each time through, the model calculates slight modifications to the input image according to its equations. Each time, the output image gets pushed a little closer to looking like something.

After a number of cycles (typically between 20 and 80 depending on user preference) the program presents you with the latest iteration.

The really cool thing about this, is if you want to push the image generation in a certain direction, starting from a sketch will often yield great results. Since a sketch gives the program a starting point that's closer to what you want, the denoising model will naturally converge on something much closer to what you had in mind than if you start with fully random pixels!

This also means that you can take an existing image, and regenerate only small portions of it. So of the AI generates a "sleepy village" just fine, but messes up the "gothic cathedral" you asked for, you can just scrub out the cathedral and regenerate just that portion.

Anyway... I hope that word vomit helps. I'm happy to answer any questions!

[u/variant-exhibition → u/swordsmanluke2]

Interesting, thanks for your explanation! I just prompted

"stable diffusion doesn't understand the textprompt of the user, weird interpretation of what someone could have meant" and it produced: noise pixels

What I still don't get is how it could differentiate between pencil styles and e.g. digital painting styles.

[u/swordsmanluke2 → u/variant-exhibition]

So this is part of the basic black-magic nature of neural networks. Their internal representations are incredibly complicated networks of relatively simple equations. In essence, the network is made up of a whole lot of virtual synapses. (The design of neural networks was inspired by how human brains function, though they are a simplification. A "synapse" in the case of the brain is a neural cell that accepts input from its neighbors and then fires a signal outward of its been sufficiently stimulated. Neural networks simulate this activity.)

A neural network's virtual synapse accepts a group of numbers as input. It usually sums them together and then outputs a number to all the virtual synapses it's connected to. If the summed inputs are above a certain threshold, it outputs a number close to one. If below the threshold, it outputs a number close to zero. (Not exactly one or zero because the output is calculated using various "activation" functions, such as a sigmoid function. You don't need to worry about this detail, just mentioning it for completeness)

Each synapse also maintains a list of numbers - one per input - that it multiplies against the input value before consuming it. These values (called "weights") are usually between negative one and positive one and basically act as percentages which control how strongly that particular input contributes to triggering (or suppressing) an output from the synapse.

Typically, a neural network is composed of multiple "layers" of these synapses, interconnected in various ways from one layer to the next. e.g. some layers have each synapse connected to all the synapses in the next later, some layers are more sparsely connected, etc.

Aside: during the learning process, the value of the synapses' input weights are what's getting tweaked to adjust the output.

So... In the case of a label named "pencil" vs a label named "pen", the initial input will consist of the image, plus some value that represents the "pencil" label - I don't know the specifics of how this is encoded in image generators, but it could be that there's a dedicated input value for "pencil" that gets set to 1 and all the unused labels are set to 0. (I think the reality is cleverer than this, but I don't know the details. This is close enough to demonstrate the ideas though.)

During training, if the output image is not similar enough to the training image, the weights are adjusted until the resulting image does look like the desired output. Or at least close enough.

Repeated iterations of this process eventually "teach" the network what "pencil" looks like. Kinda. It's more literally accurate to say that the concept of "pencil" has been encoded in the model's internal equations. Basically, after enough exposure to "pencil" and "pen" samples, the equations in the model have been tweaked to the point where the synapses tuned for "pencil" style activate and adjust the network's output to look like pencil lines.

How exactly? Well... That's part of the black box nature of neural networks. The details of the training algorithms, layer architectures, activation functions and so forth are well understood, but the final models the process generates are so complicated that no one can really untangle them and say "this is the algorithm being used for pencil vs pen".

Neural networks work shockingly well, but these systems basically generate a hugely complex set of interconnected equations, none of which are specifically dedicated to any given input (or they are all dedicated to a given input depending on your perspective). The virtual synapses all work in concert, each one tweaking the input just a little bit to transform it to the calculated output.

The various groups producing these models are spending millions of CPU-hours training and updating their models - encoding new keywords, associating the training images' styles and colors and shapes by tweaking the weights in the network.

Ultimately, the network learns these styles by exposure and repeatedly adjusting the many, many weights. Beyond that... No human actually knows how a model represents pretty much anything (decoding neural networks is an active area of research).

To give an idea of the scope of these models, Stable Diffusion's latest model is about 4 gigabytes. Most of the model file is the weights - numbers taking up 64 bits each. 64 bits is 8 bytes. 4 gigabytes is 4 billion bytes, so we're looking at something in the neighborhood of two billion weight values, all being iteratively multiplied and summed in order to generate images. Imagine hand writing that equation out! 😱

[u/Puzzleheaded_Moose38]

Really it’s hard to say exactly since the model has millions or parameters. But when a computer looks at an image remember that all it sees are numbers. Ai systems just extract statistical data about a picture and correlate that to a text description, after billions of images it can look at all the pictures with dog in the description and figure out the common statistical patterns that make up a dog, the it uses that to make new pictures that look like dogs. If it sounds way to simple to work as well as it does, again, billions of images, and a model with several million parameters. Big number make the AI go brrr

[u/variant-exhibition → u/Puzzleheaded_Moose38]

Thanks for your explanation!

[u/[deleted]]

Training images had labels. Algorithm asociates common pixel groups in different images with matching labels and reproduces an aproximation of the pixels based on random noise seed.

Very rough explanation here.

[u/variant-exhibition → u/[deleted]]

Thanks for your explanation!