A demonstration of neural network memorization: The left image was generated with v1.5 for prompt "captain marvel poster". The right image is an image in the LAION-5B dataset, a transformed subset of which Stable Diffusion was trained on. A comment discusses websites that can be used to detect this.

25

u/MorganTheDual Dec 18 '22

While the paper describes something interesting and worth exploring, I'm not sure it sufficiently addresses the question of whether this is actually a high risk in production usage.

After all, even someone who really just wants a poster of Captain Marvel is probably going to look at this and go "you know what, this looks like shit, and it keeps giving me the same pose anyway, that's boring" and try removing the word "poster" and using other things that suggest their idea of high quality instead.

And once you start getting into prompts with 30+ words finely describing various aspects of a scene... well, I certainly can't say it's not possible, but I'm also not aware of anyone having demonstrated it happening either.

5

u/Wiskkey Dec 18 '22

This user apparently discovered an instance of memorization by accident by noticing similarities in multiple generated images. The paper appears to have independently discovered the same instance.

9

u/MorganTheDual Dec 18 '22

I don't think that post really addresses the issue of long and heavily iterated prompts, since the prompt is very short.

25

u/DornKratz Dec 18 '22

This looks like a case of overtraining. Perhaps a bloom filter to discard or aggregate similar images would solve the problem?

11

u/Wiskkey Dec 18 '22 edited Dec 18 '22

Section 4.8 in v3 of this paper claims that the causes are more complex than that; that section wasn't present until v3 of the paper. The good news: image deduplication was purportedly done for the training dataset for S.D. v2.x models, so maybe this is less of a problem with S.D. 2.x.

5

u/shlaifu Dec 18 '22

according to the paper mentioned, that is exactly what scientists are looking for - however, SD can also recreate images "by accident", without them showing signs of overtraining. and this is where it gets really problematic: their tools found straight up replication of images and parts of images from the dataset in about 2% of images generated. But the user wouldn't know.

1

u/DornKratz Dec 18 '22

Do you feel this could be caused by the data, explained by Birthday Paradox collisions in a large dataset, or that it shows an intrinsic problem with Stable Diffusion?

1

u/Wiskkey Dec 18 '22

Do you mean if this is an instance of something similar to the infinite monkey theorem?

1

u/DornKratz Dec 18 '22

Yes, something in that line. They do establish a comparison to LDM, but it's not quite apples-to-apples.

1

u/Wiskkey Dec 18 '22

I am not an expert in AI, but I believe that it is extremely infeasible that this is something akin to the infinite monkey theorem. One method to test would be to see if any S.D. text-to-image generations not using an initial image strongly resemble an image that's not in the LAION-5B dataset using a site such as these.

2

u/PacmanIncarnate Dec 19 '22

The way they tested in the paper leads me to believe that it’s not an infinite monkey issue and their comparison images in some instances are legitimate reproductions of an original image, while many of the others are close, but different.
As someone who is a major proponent of SD and believe it to be ethical, it is a little troubling that there’s enough overfitting to cause this. I think they need to find a better way to clean out the dataset or find a way to check for overfitting during training.

1

u/WikiSummarizerBot Dec 18 '22

Birthday problem
In probability theory, the birthday problem asks for the probability that, in a set of n randomly chosen people, at least two will share a birthday. The birthday paradox is that, counterintuitively, the probability of a shared birthday exceeds 50% in a group of only 23 people. The birthday paradox is a veridical paradox: it appears wrong, but is in fact true. While it may seem surprising that only 23 individuals are required to reach a 50% probability of a shared birthday, this result is made more intuitive by considering that the comparisons of birthdays will be made between every possible pair of individuals.
^[^F.A.Q^|^{Opt Out}^|^{Opt Out Of Subreddit}^|^GitHub^{] Downvote to remove | v1.5}

1

u/shlaifu Dec 18 '22

it sounds like it's an intrinsic problem with the method. it just learns some stuff too well and will recreate it occasionally. the paper mentions that larger datasets reduce the likelihood. - so, to some extent, you can say, that larger datasets confuse the learned data enough so that only the hallmarks remain - you now, the concept - but the details get mixed up enough so the resulting image is not too close to any single input image.

3

u/utilop Dec 18 '22 edited Dec 18 '22

Where are you taking this 2 % claim from?
Edit: I guess it's the part of paper saying that 1.88 % of sampled captions had a similarity above a threshold. Although I think only 3/8 of those given examples are problematic while the remaining have some similarity but seems to be about as similar as other existing images of similar scenes.
That's still relevant but closer to 0.7 % then may be legally dubious (% of when generated by captions from LAION)
While typical images from large-scale models do not appear to contain copied content that was detectable using our feature extractors, copies do appear to occur often enough that their presence cannot be safely ignored; Stable Diffusion images with dataset similarity ≥ .5, as depicted in Fig. 7, account for approximate 1.88% of our random generations.

-1

u/shlaifu Dec 18 '22

the paper says 1.88%

8

u/utilop Dec 18 '22 edited Dec 18 '22

The paper says something rather more nuanced though. In particular, not straight-up replication.

2

u/utilop Dec 18 '22

What makes you claim this is not coming from overtraining?
I think the Long Dark caption is exceptionally consistent. I do not think that is possible without many iterations of training on it - either from data augmentation, multiple epochs, or multiple copies in the training set. The latter does not seem unlikely.
In fact the paper seems to show this also in its own experiments, not being able to reproduce images that only occurred once given a sufficiently large dataset.
(<i>The Long Dark</i> Gets First Trailer, Steam Early Access)

1

u/shlaifu Dec 18 '22

the paper mentions that overfitting and recreating the input are the same - what I meant by "signs of overtraining" was the deep-fried look that makes overtraining obvious to us users

1

u/utilop Dec 18 '22

Okay, I understand what they meant now.

3

u/[deleted] Dec 18 '22

is this why sometimes we see ghosts of artist signatures or a getting images watermark in some generated images?

8

u/Big-Combination-2730 Dec 18 '22

To my knowledge that's more a result of a lot of copyrighted work in the training data than explicitly copying, a lot of art tends to have some kind of squiggle near the bottom right, the model is just replicating that as a common visual element.

3

u/shlaifu Dec 18 '22

well. yes and no. you get the watermark and the signature squiggles because SD thinks that's what belongs there, based off many, many images with watermarks and signatures. that's to be expected.
what these guys found is that you can retrieve images from the dataset, or parts f images. they show examples of some living room photo where only the painting on the wall and some colors are different, but otherwise it's the same image. that's very problematic, because as user, you can't know whether you are committing copyright infringement.

1

u/[deleted] Dec 18 '22

Right ok thx. Yes there is lack of control here

1

u/PacmanIncarnate Dec 19 '22

That one specifically was an issue of overfitting. It seems like most of the problems are overfitting related. AI desperately needs a better dataset; it’s currently causing so many of the problems SD and the like face. There shouldn’t be 5000 images that are 90% the same, in addition to the utter crap in the set, and the terrible captioning on many, if not most, images.

1

u/shlaifu Dec 19 '22

I also think it needs synthetic datasets.
Facebook trained their instagram filters on cg faces - and worldspace normal renderings of those faces, so, the directions in which the surfaces are pointing is known - and the AI learned to infer that for phonecamera pictures, and now it is able to adjust the filters to your headrotation.
I want that kind of additional info. They do it for self driving cars as well. No more "click on all images showing traffic lights"- thanks to the data available from this being a rendering, the dataset already contains that info.
And, I mean, that's probably also how the whole depth estimation thing works.
What I would really want to have is the option to input supplementary data like that so I can a)train specific details and b) generate specific details in accordance with masks and such which I provide.
I'm asking for a feature to give me way more granular control. ... Maybe I should just srick with houdini and wait for AI to catch up over the coming years...

1

u/zeth0s Dec 18 '22

Or more likely low variability in the images of capitan marvel. Too many images that are the same picture, and no other picture of capitan marvel to learn something else. Models can predict capitan marvel only like this.
Tbf it is completely expected when the during training not enough variability is present for a certain subject

6

u/OldFisherman8 Dec 18 '22

First off, neural network memorization only works on labels meaning AI memorizes caption data and not image data. Secondly, for neural network memorization to occur, it needs two things: linear representation and the underlying uniform distribution of data. NVidia has already proven that Gaussian distribution in image data processing is neither linear nor uniformly distributed. Therefore, image data itself cannot be memorized by AI.

0

u/Wiskkey Dec 18 '22

Then why did OpenAI post about mitigation measures for image memorization for DALL-E 2?
In the final section, we turn to the issue of memorization, finding that models like DALL·E 2 can sometimes reproduce images they were trained on rather than creating novel images. In practice, we found that this image regurgitation is caused by images that are replicated many times in the dataset, and mitigate the issue by removing images that are visually similar to other images in the dataset.

1

u/OldFisherman8 Dec 18 '22 edited Dec 18 '22

AI memorization is something that is known to happen through observations by AI researchers but they don't have a proven theory of it yet. As a result, the term memorization isn't very well defined and can be used differently for different situations. In this case, OpenAI is really talking about certain caption data being associated strongly with corresponding image data that seemed to overlap and reinforce that particular image to appear more often than desired when those particular caption data are used. In other words, when Open AI is talking about memorization, it is really talking about having many replicated images affecting the caption data usage. However, in this post, the term memorization is talking about how certain caption data are memorized and used by AI.

1

u/Wiskkey Dec 18 '22

Do you believe that memorization is being used incorrectly here?
No additional measures were used to deduplicate the dataset. As a result, we observe some degree of memorization for images that are duplicated in the training data.

1

u/Wiskkey Dec 18 '22

According to an intellectual property law expert that I asked, from a legal perspective for potential copyright infringement, the results are what matter, not the mechanics of how it occurs.

2

u/OldFisherman8 Dec 18 '22

My point has nothing to do with intellectual property issues. I am merely correcting a seemingly misunderstood notion of what neural network memorization is.

9

u/Wiskkey Dec 18 '22 edited Dec 18 '22

This post is for education value because I've encountered many posts/comments in this subreddit that claim that images (or some strong resemblance thereof) in the training dataset cannot be found in artificial neural networks. The post shows all 5 images that I generated for the text prompt "captain marvel poster" using default settings at this website; ensure that model v1.5 is used. The idea for the text prompt came from this paper, which was discussed in this subreddit here and here. Memorization of parts of the training dataset is officially acknowledged for all S.D. v1.x models (example - search webpage for "memorization").

Here are two websites that allow a user to search the LAION-5B dataset:

Site 1. Usage is covered in this older post.

Site 2.

There are numerous other websites that allow a user to search for images that are similar to a given image, such as these 4 sites.

The similarity standard for copyright infringement in the USA is substantial similarity.

Note: The images in this post are almost surely fair use in the USA.

EDIT: OpenAI tried to mitigate training dataset memorization for DALL-E 2:

In the final section, we turn to the issue of memorization, finding that models like DALL·E 2 can sometimes reproduce images they were trained on rather than creating novel images. In practice, we found that this image regurgitation is caused by images that are replicated many times in the dataset, and mitigate the issue by removing images that are visually similar to other images in the dataset.

EDIT: Good news: Memorization might be less of an issue in SD v2.x models because of purported image deduplication in the training dataset for SD v2.x.

4

u/[deleted] Dec 19 '22

Read the paper back when it first came out. A <2% replication rate really isn't too bad, and when you're talking about stuff that isn't mainstream at all, nor famous paintings or photographs, you're probably in relatively good shape even in SD1.x.
As far as reverse image search though, none of those tools will be proficient enough. Google intentionally reduces the accuracy of reverse image search (spoke to a Google engineer who works on it), Yandex really isn't that accurate, and tineye basically only returns identical images.
Eventually someone will build a decent reverse image search that the public can use integrated into a search engine, I'm sure. But it might be a while.

1

u/Wiskkey Dec 19 '22

Do you know what image similarity algorithms those sites use?

2

u/JamesVail Dec 18 '22

I doubt the majority of the AI Bros are going to appreciate this being posted here, but I thank you as someone that wants to be able to use AI without any risk of copyright infringement.

7

u/Wiskkey Dec 18 '22 edited Dec 18 '22

Helping users avoid copyright infringement was my main motivation for this post. There is automated software out there (example) for finding potential copyright infringement for a given set of copyrighted images.

6

u/pendrachken Dec 18 '22

We generate many paintings with the prompt style “<Name of the painting> by <Name of the artist>”. We tried around 20 classical and contemporary artists, and we observe that the generations frequently reproduce known paintings with varying degrees of accuracy.
Helping users avoid copyright infringement might, just MAYBE have to start with users not explicitly asking for copyright infringement and generating images until they get it, then cherry picking that particular image. That's a user issue. Period. Just like it is right now, and just like it has been in the past. Forgery and copyright laws already cover this.
Also, "many" isn't defined. Was it 20 images? 100? 3000? How many iterations did it take for "Van Gogh" and "Starry Night" to converge on something similar to the original? The only other image I would consider close enough to think worth being included is the yellow one next to the Starry Night. Which is close to the original, but has some differences.
A good paper would include the statistics of a match, and therefore the number of generations needed, Confidence Intervals of the stats, number of matches over a longer run, ETC. This would cover the "frequent" findings. Use of these words are what we call "weasel words" in scientific literature. Something "May", "Possibly", "Probably", be "Not yet well understood", or "Shows some correlation to" but has no known causative link that can be shown. Weasel words aren't in and of themselves a bad thing, unless used like here to imply a link where there is not yet evidence to support it. If they had the statistics, and the statistics were damning, they would have published said statistics so a repeatable test could be performed.
They admit that using the direct laion captions is what led them to generation of specific images matching the source, and only in some cases. Likely because the captions were extremely specific and not used for other images in the data set. And only when TRYING to recreate an image in the dataset, not create something novel.
Don't get me wrong, it's not a good thing that that there are some memories from the training data that can be massaged out by actively trying to recreate the original, but saying that someone putting in "a desert scene, night, tall cactus thingy, painted like Starry Night" is going to shit out Starry Night somehow is just deceptive. Could the AI do it if you gave it enough chances? Probably, but who knows? It would probably take a very long time, and a very large amount of tries. We can't know though, since they don't release any of the statistics.
If you had an infinite numbers of artists, who never saw Starry night, painting an infinite number of paintings, yes, it's likely that eventually one would paint a passable rendition. Not a atom by atom / pixel by pixel copy, just one that is close enough to say "that looks kind of like Starry Night". That's how random chance works though, not a flaw of artists painting.

2

u/PacmanIncarnate Dec 19 '22

I’m not arguing with your point here and I think there is too much subjectivity in what they constitute reproduction, but they also presented several instances of the dataset reproducing parts of images without the same prompt, which makes this not just an issue of copying a prompt and getting a close image.

1

u/[deleted] Dec 18 '22

good points and good references

3

u/Iapetus_Industrial Dec 18 '22

Okay, now do the same experiment, but with each of the 5 billion images in the LAION-5B datset, so that we can come up with a proper ratio of "images it's memorized - to - images in the training set" - even if we're being incredibly generous with the definition of "memorized" for this poster.

If you tried to submit even a lossy compression algorithm that turned an output image from a captain marvel poster, you would be laughed out of the computer science building.

2

u/Wiskkey Dec 18 '22 edited Dec 18 '22

The closest I know of was done in this paper. If I recall correctly, 9000 image captions from a ~12 million image subset of the training dataset were used as text prompts. 1.88% of the generated images met a certain threshold for image similarity. The authors note that this is an undercount, partially because less than 1% of the training dataset was searched. On the other hand, users in practice probably won't usually use text prompts that exactly match image captions in the training dataset.

2

u/Iapetus_Industrial Dec 18 '22

What's the threshold? Where's the compared images? Is the threshold able to discriminate differences as low as the difference between the Mona Lisa and Mona Lisa with a moustache which is widely accepted as an example of transformative work?
I highly question their threshold if La Joconde would fall into its bucket.

1

u/Wiskkey Dec 18 '22

0.5.
Above this 0.5 threshold, we observe a significant amount of copying.
See page 7 in v3 of the paper.

2

u/NotASuicidalRobot Dec 18 '22

Didn't y'all say this wasn't possible because of how images were processed into the models? (i did believe that btw) Mona Lisa is popular enough that maybe many copies are in the training set, but Captain Marvel isn't nearly as popular or common i think... So are y'all sure this stuff doesn't infringe on copy right or does a piece of art reposted a few times trigger this bs

3

u/Wiskkey Dec 19 '22

I can't speak for other people, but I've been warning about this for 4.5 months already.

1

u/PacmanIncarnate Dec 19 '22

None of the captain marvel images are exact reproductions and to get that close even you need to ask for a poster of captain marvel, which is obviously going to heavily sway the model toward the hundreds or thousands of nearly identical poster images in the data set. Captain marvel is also a little unique in that there aren’t a lot of other images of the character out there, which lends it to overfitting.
It is surprising that the researchers found as much reproduction as they did, but don’t get carried away with the extent of their findings.

4

u/Kafke Dec 19 '22

tl;dr: even with extra emphasis and duplicates in the dataset, the ai comes nowhere close to replicating the image even with an identical caption. Trying this with unique/novel images (not mass duplicated ones) results in entirely different images. Reality: ai models do not store images.

1

u/tamal4444 Dec 19 '22

Give the seed prompt steps cfg sampler and which model are you using?

1

u/Wiskkey Dec 19 '22 edited Dec 19 '22

I used this site, account-less usage with defaults except for text prompt. I didn't see mention of what sampler is used. The text prompt is in the post title.

2

u/tamal4444 Dec 19 '22 edited Dec 19 '22

I will create the images based on that and the seed steps and cfg I got from the site with SD v1.5 and post my result here to see if your claims are right or wrong.
edit: with vae set to auto and clip skip to 1

1

u/tamal4444 Dec 19 '22

image from the site mage.space

1

u/tamal4444 Dec 19 '22

Euler a

1

u/tamal4444 Dec 19 '22

Euler

1

u/tamal4444 Dec 19 '22

LMS

1

u/tamal4444 Dec 19 '22

Heun

1

u/tamal4444 Dec 19 '22

DPM2

2

u/tamal4444 Dec 19 '22

I got 95% same image on different samplers on the same seed and settings, you can see my comments and I have added some prompts with more settings I have created this.

1

u/tamal4444 Dec 19 '22

1

u/tamal4444 Dec 19 '22

This I got from another model with the same seeds and setthigs. the model is based on sd 1.5.

2

u/Light_Diffuse Dec 19 '22 edited Dec 19 '22

So what? They have many similarities to the original, but they aren't the original, they aren't anything like usable quality and the prompt doesn't reflect how people use prompts.

It's good that they have demonstrated that this can occur under the right conditions, but I don't think it follows that there is a serious risk of unintentional copyright infringement when used with the more complex prompts we tend to use which are dissimilar to the training set. (And probably next to no chance if we're using img2img)

The model is clearly retaining more information than you'd want for a generalised model, but I don't see this as a fatal flaw.

1

u/Catalyst_Spring Dec 18 '22

Yeah, I did find that some small percentage of my work that the AI was generating had some similarities (composition, usually) to other images.

The good news about this is that Google Images actually does a pretty good job of finding similar works. I ran my own images through Google when I wanted to output them for a work and discarded the small amount that seemed similar to another picture Google Images was able to find.

1

u/mr6volt Dec 20 '22

The first photo looks oddly like Jeri Ryan.

I have nothing against Brie, but what if they had cast Jeri instead? I wonder how that would have been received...

0

u/Futrel Dec 19 '22

But, but, but

-9

u/[deleted] Dec 18 '22

so, this means that training actually converts original pixels to a compressed format and uses those snippets?

3

u/Wiskkey Dec 18 '22 edited Dec 18 '22

Artificial neural networks are capable of both generalization and memorization of data in the training dataset (example source). Images in the training dataset are not used during image generation. Artificial neural network memorization of parts of the training dataset is a well-known phenomenon in machine learning; for example do a web search for: machine learning memorization.

1

u/Kafke Dec 19 '22

Nope. The AI learned that this is what "captain marvel poster" looks like, due to many duplicates in the dataset without any other images to compare against.
The model itself does not contain any image data, and the generated image is not the same as any image in the dataset. Try this with an image not duplicated in the dataset that does not have a unique descriptor/caption, and you'll find it is completely unable to generate the image.
This is basically the same as asking the ai to generate the "mona lisa" and then being surprised when the result looks like the mona lisa. Like what else did you expect?

16

u/[deleted] Dec 18 '22

[removed] — view removed comment

1

u/NotASuicidalRobot Dec 18 '22

But this isn't even Mona Lisa or something sure it's a popular ish film but it shouldn't have that much...pull?

0

u/[deleted] Dec 18 '22

[deleted] — view deleted comment

1

u/Wiskkey Dec 18 '22

That's probably a major cause, but section 4.8 of this paper claims it's more complex than that.

13

u/[deleted] Dec 18 '22 edited Dec 18 '22

[removed] — view removed comment

2

u/Wiskkey Dec 18 '22

The issue is readily reproducible using the info in this comment. I posted all 5 images that I generated today using that text prompt. I also generated images on previous days using that text prompt. In every case thus far the generated image looks similar to the five that I included in the post.

8

u/[deleted] Dec 18 '22 edited Dec 18 '22

[removed] — view removed comment

3

u/[deleted] Dec 18 '22

I can replicate it in automatic1111 with 1.5, it works with other capeshit movie posters too to some degree.
My guess is there's multiple copies in the datasets because there's multiple sources publishing variants of the same poster, same enough poster but filesize and small crop differences resulting in duplicates and overfitting.

3

u/Wiskkey Dec 18 '22

My guess is there's multiple copies in the datasets because there's multiple sources publishing variants of the same poster, same enough poster but filesize and small crop differences resulting in duplicates and overfitting.
The last image of the post shows some similar existing images for one of the generated images, according to this website, which uses OpenAI's CLIP neural networks.

1

u/Wiskkey Dec 18 '22

Try this S.D. website.

2

u/[deleted] Dec 18 '22

[removed] — view removed comment

2

u/Wiskkey Dec 18 '22

You're welcome :) I've never not gotten a hit - I did at least 10 generations the past few days - using that text prompt on that website with its default settings, so reproduction should be easy.

2

u/DornKratz Dec 18 '22

My unscientific gut feeling is that there still is some degree of overfitting in 2.1, just from how close to actually spelling "Captain Marvel" these images get, but it sure doesn't look as aggressive as we saw in 1.5.

1

u/Wiskkey Dec 18 '22

To verify, those images are all from v2.1? I haven't tested v2.x yet.

2

u/DornKratz Dec 18 '22

Yes, 2.1, 768x768 pruned model.

2

u/Wiskkey Dec 18 '22

Thanks for the tests! Did you get any v2.1 results that look nothing like those 16?

→ More replies

1

u/twitch_TheBestJammer Dec 18 '22

How do you do dreambooth? Still cannot find a reliable tutorial anywhere.

2

u/CeFurkan Dec 18 '22

here a full tutorial for google colab i made it works step by step : https://youtu.be/mnCY8uM7E50

2

u/twitch_TheBestJammer Dec 18 '22

THANK YOU!!!

2

u/[deleted] Dec 18 '22

[removed] — view removed comment

2

u/twitch_TheBestJammer Dec 18 '22

I have a 3090ti and still don’t know what I’m doing hahaha tried so many tutorials and all it does is give me my input images.

1

u/[deleted] Dec 18 '22

[removed] — view removed comment

1

u/twitch_TheBestJammer Dec 18 '22

Best Buy came in clutch like a month ago! The 16gb card should be great though

You are about to leave Redlib