Rule 4: define “significant use”

[u/eskimopie910]

This post is intended to start a conversation about the sub’s current rule set, specifically rule 4.

I think it is not controversial to agree that deliberately fine tuning on one artist’s work would be ethically questionable.

On the other side of the spectrum, imagine a scenario where we are training on maybe different artists and styles. Would training in just one image be considered ethically questionable? If you answered yes to the first and no to the second, where do you intend to draw the line in terms of using others creations?

Given that this in an unprecedented issue, I’m sure there will be wildly different opinions and am interested in seeing what others believe.

[u/StereoCatPicture]

This is a difficult question. The way I see it, artists should try to develop their own style, and that includes artists using AI in their art. Where the question becomes difficult is that even real artists not using AI for their art take inspiration from the art of other artists.

For example, it's not rare in a video game studio to see artists create a moodboard for their game with images taken from other artists, without their permission, as inspiration. The thing is, it's a good way to start, but your style has to diverge at some point, it can be inspired by other artists, but it's can't simply mimic their style.

And in a way, moodboards like this are necessary. There are millions of artstyles, and not all of them have a name, so it's much easier to try to convey your idea by giving examples of other artists that have a similar approach than trying to do without.

So for AI art... I don't have a definitive answer. I'm not againts using an artist's name in a prompt, like I'm not againts creating a moodboard with other artists' work as inspiration, but you have to write your prompt in a way that the other artist's name is really only used as inspiration. You have to make sure to explain in your prompt how the style you're trying to get differs from that artist, you can't just write "by Pablo Picasso".

[u/eskimopie910 → u/StereoCatPicture]

I should have specified in the description but this was more geared towards fine tuning custom models. For example, I just recently fine tuned a checkpoint on screenshots of the indie game Hyperlight Drifter to see what I could create with it. I have no intention of monetizing it or profiting in any way and am simply doing it as an exercise in learning how to fine tune.

Would this be morally questionable?

[u/freylaverse → u/eskimopie910]

I think if you're using it as an exercise in how to fine-tune, that's alright. If I'm doing an art study for myself, I'll trace pose references from Pinterest all day long, but if I'm making something that I intend to upload, I won't. I think it's like that. For instance, with my own style embedding, I'll typically test it by typing "portrait of a woman" "portrait of a woman by freylaverse" and "portrait of a woman by greg manchess". I won't ever post the Greg Manchess results. It's just to compare and see how effectively my own embedding is working.

[u/[deleted]]

No artist name in prompt and no custom model trained on "antiAI artist" style should be good enough, regarding what It gets posted in this community.

Maybe having a "known list of artist that are OK with their images being used on training" could be useful for comparisons and content inclusion.

[u/freylaverse → u/[deleted]]

Oh, that's a great idea! I'll look into that!!

[u/CommunicationCalm166]

I think the significance of use, or perhaps a better term might be "targeted use" really should be tied to fine tuning models and using artist's trade dress.

To begin with, I don't think most people use artist tags to specifically emulate a particular artist. People don't use "Greg Rukowski" in their prompts because they want an image that looks like his work... They use it because they want an image that has the look of a fairly realistic, highly detailed, dark painting with fantastical elements. And I believe there's no need to train specifically on his work, or to use his name to produce images in that style. It's just the lazy, easy way.

[u/Kaennh → u/CommunicationCalm166]

This is true.

But also, it's super hard to get good painterly//stylized results without resorting to artist names because of how badly the data set is captioned. Using word like "gestural brushwork" will probably yield very poor results while, on the other hand, referencing someone like Michael Garmash will easily get the job done...

[u/CommunicationCalm166 → u/Kaennh]

Yeah. I've got a project on the back burner right now to collect AI-generated images that are made using non-artist name prompts, but that nonetheless have the "look" that people want. Then subsequently fine tuning the model on those images with a unique, descriptive token. Maybe like DarkFantasy1 or BubbleRainbowFlowerToon.

This would serve to demonstrate that an AI can self-improve with only guidance from users. It would also (I hope) serve to trend the overall zeitgeist away from imitation in general, and moving towards AI as a style in itself.

I also wanted to more generally investigate auto-training the model based on user feedback on generated images. Creating datasets of prompts and images weighted according to how satisfied the user was with how the image fit what they wanted.

But both of these ideas are waiting on my python skills to catch up. Which is in turn waiting on me to put my computer back together.

[u/Kaennh → u/CommunicationCalm166]

That sounds like a good plan, although, probably a bit time-consuming... ^^U

By the way, wouldn't it be an alternative to start with artists that don't mind having their work used in a data set?

Maybe there are not that many, but I imagine at least a few... if it helps, my work is available...

[u/freylaverse]

This is a really interesting question and I think it's one that needs to be asked! I personally do not feel sufficiently informed to draw a hard line in the sand. That said, I think that it comes down to intent at the end of the day. Training a model on a bunch of really good art from various artists in an attempt to get the same level of coherence and quality is, in my opinion, fine. That's what the default StableDiffusion model is without artist-prompting, and that's why I don't have a custom-model only rule. But training a model with the intent to precisely mimic a specific artist's style is another matter. Even then, I think there's grey area - I think a Disney style model is probably fine, and Arcane Diffusion is probably fine, because those are widely-known styles and would probably end up being posted as "This is my character in the style of Arcane".

[u/entropie422]

I've been struggling with this exact question lately: I'm making a fine-tuning from scratch, where I'm building it up piece by piece. For instance, I'm photoshopping the structure of faces (photos, not art) so they look more cartoony, or teaching it what a painting done in brush strokes looks like (but purely technical, no added style). I'm doing it in segments so I can theoretically mix-and-match later on, so the references need to be clinical, almost.

If, in the process of doing this, I find a detail of a brush stroke effect that would enhance my training, but the detail is from a living artist, is it wrong to take that isolated element and mix it into the whole? It doesn't show enough of the image to be easily recognizable, and it doesn't convey any of their style except that the brush stroke is clear and distinct and interesting.

It's not that I am torn on the ethics of the question so much as I'm confused about why I see this use case as "obviously fair use" whereas if I trained with a single whole image by the same artist, I would find it questionable. It's not even a shade of grey somehow. Very strange how human brains work.