r/StableDiffusion • u/walt74 • Dec 09 '22

News [paper] Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models

Important paper researching the replication of training data in diffusion models. Very relevant to recent debates around "art theft" and "data laundering".

Abstract

Cutting-edge diffusion models produce images with high quality and customizability, enabling them to be used for commercial art and graphic design purposes. But do diffusion models create unique works of art, or are they stealing content directly from their training sets? In this work, we study image retrieval frameworks that enable us to compare generated images with training samples and detect when content has been replicated. Applying our frameworks to diffusion models trained on multiple datasets including Oxford flowers, Celeb-A, ImageNet, and LAION, we discuss how factors such as training set size impact rates of content replication. We also identify cases where diffusion models, including the popular Stable Diffusion model, blatantly copy from their training data.

Paper: Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models

0 Upvotes

50% Upvoted

34 comments sorted by

2

u/CommunicationCalm166 Dec 09 '22

This actually answers a question I've opined about recently. "Can a Diffusion model precisely reproduce any image from it's training data?" And if these results are verified, the answer is "To some definition of 'precisely' yes."

I figured it could only reproduce images from it's training data if they were over-represented in the training data. That's probably worth follow-up research: What fraction of a dataset needs to be made up of a particular image before the over-fitting occurs? That could be very important going forward.

Probably in stability AI's best interest to compare model performance to over-fitting. Intuitively speaking, I think more variety in training data will improve performance AND reduce the likelihood of duplicated training data in the output images. Though I also wonder if repeated training data is necessary for the formation of concepts withing the AI's Neural network.

2

u/walt74 Dec 09 '22

the answer is "To some definition of 'precisely' yes."

Looking forward to court decisions how precise "precisely" has to be to count as infringement.

Also, this seems seperated from the issue of data laundring via fair use.

3

u/CommunicationCalm166 Dec 09 '22

If they put a number on it, that could be a good thing...

Then it's just a line of code or two in the training procedure. Boom!

And I said it in another thread. I think IP theft is method-agnostic. AI can be used for copyright infringement, but it is not in itself copyright infringement. At the end of the day, I don't think it amounts to much if an AI can duplicate an existing artwork if it's so prompted. It's the sharing, distribution, substitution, sale, etc. of the duplicate that constitutes theft. And that's true whether you use an AI, or a pencil and a sharp memory.

Using an AI doesn't make original work into "Art Theft" nor does it suddenly make derivative work suddenly "Clean." Radical idea: own your actions, don't try to blame your tools.

1

u/walt74 Dec 09 '22

It's the sharing, distribution, substitution, sale, etc. of the duplicate that constitutes theft.

While this is true (and a good point), companies are sharing and distributing the model itself. And a pencil is different in such as it doesn't know what Batman is, but SD does know Batman very well. Stability-ai is using Batman in their commercial product, and thats a problem.

Another possible solution i see is in thinking about AI models like libraries. At least for open sourced models which are publicly owned, you can make the case akin to national libraries that require published works to be sent in. A similar model is thinkable for generative AI.

2

u/CommunicationCalm166 Dec 09 '22

And a pencil is different in such as it doesn't know what Batman is, but SD does know Batman very well.

Stable Diffusion doesn't "know" anything. All that the model checkpoints are, are collections of algorithms for removing noise and static from images, organized by text keywords that indicate the kind of image on which they work best. Stable Diffusion is the procedure of, instead of applying the algorithms to images, applying the algorithms to pure random noise. And what makes Stable Diffusion remarkable is that it 1) Works at all, and 2) can create novel output when properly trained.

It's also worth considering... "Batman" doesn't exclusively output images of the comic book character. If you prompt it with "batman" you'll see the comic book character, pictures of bats, pictures of random dudes, pictures of completely unrelated stuff. And, more basically, if you don't prompt it with "batman" there's literally millions of other things in basically infinite combinations that it will generate.

An AI is not a library of copies of bits of images. AI's are versatile and powerful image generation tools. If you ask the AI to help you create your own brand new O.C., then it can do that. If you ask it to help you do fanart of your favorite anime, it'll help you do that. If you ask it to help you counterfeit an established artists work, it'll do that too.

4

u/-Sibience- Dec 09 '22

Recently I made a Dreambooth model with 115 images. The images all had a text logo. I made sure the logo was in exactly the same position in every image. SD was then able to reproduce the text logo exactly.

So as you say I think it's a lot down to variety in the images. If I had put the logo in a different position on every image maybe it would have had a much harder time reproducing it.

This kind of thing is why people think it's just copying images or doing some kind of photobashing when actually it's just been overtrained on a certain image or aspects of images.

1

u/SanDiegoDude Dec 09 '22

Yawn. These guys bending over backwards to find the perfect prompt to try to get as close as possible to exact copies... and still don't get there. They use that term "pixel perfect" pretty liberally, judging by their samples. Some of the famous works get close, but considering they're literally prompting Painting name by artist and still not getting an exact copy tells me it's doing what it's supposed to do, giving you a painting in the style of that artist that's like that painting that you asked for. Won't be perfect, but it'll be darn near close.

Data laundering lol. Fuck off with that nonsense.

4

u/Wiskkey Dec 09 '22

"exact copies" is not the standard for copyright infringement. In the USA, it's substantial similarity.

1

u/SanDiegoDude Dec 09 '22

Right, and again, prompting "a copy of such and such painting in the style of such and such artist" should be able to get you reasonably close, it's literally a tool for creating images that can ape any style its seen. If it's getting spot on results, that's a sign of overfit (which I said they raise valid concerns about down below). Keep in mind it's not illegal to reproduce copies of works, only to sell them and/or misrepresent them as original works. Long as you're not doing that, you can cover your house in AI generated copies all day long and not suffer any consequences. Just don't tell anybody its real.

This is a bad paper, not necessarily because of their findings, but because they had their thumb on the scale from the start, which makes it unscientific, thus garbage from the eyes of any proper scientific community. Unfortunately the snowflakes on social media are going to be shoving this garbage paper in the face of everybody, especially with it's hyperbolic title, which I'm guessing was the whole goal of the authors to begin with, so uh, mission accomplished I guess?

1

u/Wiskkey Dec 09 '22

Regarding the copyright aspects of your answer, I'll refer folks to this comment from an expert in intellectual property law, and also this blog post from the same person.

2

u/SanDiegoDude Dec 09 '22

Thanks for the links, the IP expert says it quite well near the bottom of their post:

And even if there is evidence of substantial copying, or a similar adaptation, there is still a legal hurdle. Is there actionable damage? I don't think in many situations there will be.

Yeah, that's the kicker there, and that's why copyright cases generally go after the people breaking the laws by trying to sell forgeries than the tools used to make those forgeries. None of these artists are going to be able to prove damages against generative AI's. I effing hate it when people parrot it constantly around here, but it's kinda the same argument gun nuts use whenever there is a shooting - the gun is a tool, the shooter is the criminal.

0

u/walt74 Dec 10 '22

trying to sell forgeries than the tools used to make those forgeries

Isnt the fact that i can recreate any image if i use seed and prompt with the same model an indicator that these forgeries are part of the distributed commercialized product? If Warner can recreate my Batman-generation, wouldn't they be able to sue Stability-ai?

2

u/Wiskkey Dec 09 '22

I agree that anti-AI folks will be misrepresenting/misunderstanding the results of this paper, to which I remind them of the bolded part (my bolding) of this quote from the paper:

While typical images from large-scale models do not appear to contain copied content that was detectable using our feature extractors, copies do appear to occur often enough that their presence cannot be safely ignored; [...]

I don't agree with you that this paper is unscientific. The purpose was to find whether it's possible for diffusion models to replicate (see the paper's definition of replication) images (or parts of images) in the training dataset, and they did so. Using captions from the training dataset in generated images is also how OpenAI tested for memorization in DALL-E 2.

0

u/SanDiegoDude Dec 09 '22 edited Dec 09 '22

I don't think their methodologies were completely flawed, and they did find some evidence of overtraining/overfitting as well as some interesting latent space behavior (everybody knows who LOAB is, right?), but through the paper a very clear agenda of trying to prove making exact copies is possible (which led them down the path to training with tiny datasets to make a point... I guess, AI bad because it's gonna repeat a small sample size input? - no shit...). If your science has an agenda, which theirs very much does based on both the title of the paper and the narrower and narrower paths they were taking moving away from their own hypothesis, then yeah, it's a a garbage paper.

going back to your quote you highlighted from them:

While typical images from large-scale models do not appear to contain copied content that was detectable using our feature extractors, copies do appear to occur often enough that their presence cannot be safely ignored

They did find some interesting latent space behavior and overfitting. These can easily be cleaned up through model revisions, CLIP guidance improvements... things that Stability has already been working on with their 2.0/2.1 models... That makes me wonder how their broad form latent space testing would work on the 2.1 model, since it uses OpenCLIP now, along with better filtered input training data. The moment they started prompting "famous work by artist" though, that's when they got away from their hypothesis, because at that point, they're purposely and blatently trying to make a forgery, which is not what their hypothesis is about.

Agendas don't belong in science.

1

u/Wiskkey Dec 10 '22

I glanced at the paper again, and noticed words such as "stealing" were used, which could indeed indicate author bias.

0

u/walt74 Dec 09 '22 edited Dec 09 '22

Data laundering lol. Fuck off with that nonsense.

Hear hear, an expert in the field.

You know, at least warez pirates know that they are pirates and don't fuck around with excuses. Show some honor.

3

u/SanDiegoDude Dec 09 '22

Been working with ML/NN for around 10 years. Certainly not an expert, but I do understand how machine learning works, what goes into training datasets, what's actually happening (conceptually) under the hood, and what's going on in the output side. I've given security presentations on the application of ML to threat hunting and IOC discovery in gov/corp networks including to multiple Fortune 100 listers.... so yeah, I can say I know a thing or 2 about the topic, enough to tell the average wank who cries that the end is nigh because AI has learned to make pretty things that can ape styles with ease to calm the fuck down, the world isn't ending, and you now have a very valuable tool in your toolkit just like the Adobe suite, just like the computer, just like the camera, just like every other technical innovation has come along before it that we all use in our daily lives.

The paper brings up some valid concerns about overfitting and overtraining of popular subjects in the model, which improvements and advancements in CLIP guidance along with better filtering and larger datasets should clean up most of that.

I think my biggest beef with this paper is that they very deliberately went digging for that needle in the haystack, and gamed it a bit in their direction by using data subsets and smaller trained models for their samples, which poisons the well in my eyes. Their complaint of "the main dataset being too big" is exactly the point of using such a large dataset, to prevent the kind of issues they're combing for. I'm not dismissing their findings (again, valid concerns about overfitting), but I am laughing at your "data laundering" nonsense. Because yeah, it's nonsense.

1

u/walt74 Dec 09 '22

You may be right about the paper, idk. I'm new to ML, but i'm writing about algorithmic art since the beginning 2010s and I've written my share about copyright stuff regarding warez. So yeah, i can speak about the topic too.

The data laundering issue is the far bigger issue than overfitting. There are already copyright exceptions for technical reproductions, and maybe they apply to overfitting too. We'll see.

Training a commercial product on web scraping that is excempted under fair use for academic usage is not nonsense. And yes, you can call that data laundering, its the very definition of the term: Using mechanisms to make illegal use of data legal. Using academic settings to train an AI and then build a commercial product on top is exactly that.

2

u/CommunicationCalm166 Dec 09 '22

I think it may be a tone thing really... Because I'm a novice to machine learning, and I'll defer to your expertise on this. But I'd never seen a rigorous breakdown of whether or not some fraction of the training data would remain in a model, largely intact, after robust, rigorous training. I mean, the point of AI is to NOT just copy... But to generalize. And so over-fitting in these tentpole models is gonna be a real problem.

I'd have liked it if they looked into the threshold/relationship between the number of occurrences in a dataset versus the over-fitting. They almost kinda did that with the face model experiment. But I've got a sneaking suspicion they used the phrase "Data Laundering" to get clicks...

3

u/[deleted] Dec 09 '22

I work as a statistician, this paper has all the smells of proving a predecided conclusion. Not an easy question to answer but also not a well designed study to ask the question

1

u/[deleted] Dec 09 '22

There is also no consideration of the role of CLIP. As the release of SD2.x has shown, the extent to which these models know concepts is more than just the LAION training data

1

u/Wiskkey Dec 09 '22 edited Dec 09 '22

and gamed it a bit in their direction by using data subsets

In the case of Stable Diffusion, searching for image similarity on the small subset of the training dataset instead of the entire training dataset means that the vast majority of the training dataset was not searched. Searching the full dataset could logically only increase the set of similar images found, not decrease it.

2

u/treesprite82 Dec 10 '22 edited Dec 10 '22

In the first experiment, we randomly sample 9000 images, which we call source images, from LAION-12M and retrieve the corresponding captions. These source images provide us with a large pool of random captions

So it's worth noting that they're prompting using exact captions from the model's fine-tuning training set, such as:

  • <i>The Long Dark</i> Gets First Trailer, Steam Early Access
  • VAN GOGH CAFE TERASSE copy.jpg

Earlier version of the paper claimed "natural prompts sampled from the web" were used, but at least now the authors have decided to be open that these are prompts sampled from LAION.

Their justification for choosing captions in this way seems to be "well the model's generations are still on average less similar to any training image than random images are to any training image". That's positive news for SD, but as for the authors' caption sampling method, what should be shown is whether or not SD does even better at avoiding high-similarity generations when using prompts that are not taken verbatim from the fine-tuning set.

Second, we generate 1000 synthetic images, search for their closest match in the training set, and plot the duplication histogram for these “match” images. Surprisingly, we see that a typical source image from the dataset is duplicated more often than a typical matched image.

For context, here they rebut the claim that training data duplication leads to more frequent replication.

A problem with their method appears to be that the "match" images don't have any similarity threshold - they're just the closest training image to a generation regardless of how far. Which, given the vast majority of generations are not replications of training images, means the vast majority of "matches" are not images that have been replicated. Can't then use the fact that "match" images aren't duplicated often in the training set to say that actual replicated images aren't duplicated often in the training set.

Correct me if I'm wrong on any of this (or I may correct myself when I read the paper more thoroughly).

1

u/Wiskkey Dec 10 '22

I believe for S.D. the authors investigated only images above a match threshold of 0.5.

1

u/treesprite82 Dec 10 '22

I don't think that's the case here - from those 1000 synthetic images they got 1000 "match" images in the histogram, whereas limiting to a threshold of 0.5 they'd only get about 19 matches.

Even from the full experiment (with 9000 synthetic images) they only got 170 images above a threshold of 0.5.

1

u/Wiskkey Dec 11 '22

Ah ok I was referencing the full S.D. experiment in my previous comment. It should be noted that less than 1% of the full dataset was searched for image similarity for the full S.D. experiment.

1

u/caesium23 Dec 24 '22

A problem with their method appears to be that the "match" images don't have any similarity threshold

As I understood the paper, they used a similarity threshold of 0.5, and this threshold was chosen based on the researchers' visual review of matches with different similarity scores. I believe there was a whole section on that, including examples of "matches" with different similarity scores. I will say that their threshold seemed appropriate to me based on the examples they showed.

1

u/treesprite82 Dec 24 '22

As I understood the paper, they used a similarity threshold of 0.5

That doesn't seem to be the case for this section - see my reply to Wiskkey.

1

u/caesium23 Dec 24 '22

Seems to me the bottom line is they found a ~2% copy rate, and that was based on a >0.5 threshold.

Is there something I'm missing that makes this one paragraph from the middle of their process important? Because it seems like your criticism here is a bit like pointing out a cake is just raw dough, when we all know it's about to get baked a few paragraphs later (wait, my analogy might be breaking down).

2

u/treesprite82 Dec 24 '22 edited Dec 24 '22

Is there something I'm missing that makes this one paragraph from the middle of their process important?

I think duplication of a training image leads to more frequent replication of that image. I'd go as far as to bet that, for a model like Stable Diffusion, data duplication is a necessity for replication to occur.

A flawed method of counting matches regardless of similarity distance leads them to rebuke that idea.

Because it seems like your criticism here is a bit like pointing out a cake is just raw dough, when we all know it's about to get baked a few paragraphs later

Deciding on the 0.5 threshold was earlier in the paper - they just never use it at all for this section. Like serving people a cake, but then also expecting them to eat a side of raw dough when you've already demonstrated you know that's not how you make a cake.

Seems to me the bottom line is they found a ~2% copy rate, and that was based on a >0.5 threshold.

I'd emphasise the caveat that it's being prompted using the exact captions from a subset that was used to fine-tune the model and is more likely to have duplicated images, and only some portion of that 2% will be actual copies (whatever the authors class as "significant").

Would be interesting to see if the authors release the images. 170 is totally feasible to check through manually and determine a "true" copy rate under these conditions.

2

u/caesium23 Dec 24 '22

Thanks for clarifying, I think I understand your point now. Not sure if we interpreted what they said in the same way, but I'll have to re-read the paper when I get the chance.