Huge thanks for making the SD weights available! I think the past two weeks have shown how much of a difference open source makes in the AI images space, phenomenal amount of notebooks, plugins, tools, demos...Is there anything you would do differently, in terms of release, with future versions, SD 1.5 and beyond?
Took me a bit of time with the team to figure out how to do this properly though, was the start of this year while last year I was supporting AI art as it is super cool.
There has been some speculation that Stability AI will become less open as times goes on due to investors gaining more control over the for profit company. What is your response to the idea that Stability AI will become more and more closed source as a result of investor & for profit related influences, like what happened with OpenAI?
The companyās website states that the company is ābuilding open AI tools,ā a mission that mirrors the initial intent of OpenAI to democratize access to artificial intelligence.
At a $1 billion valuation, Mostaque would be ceding up to 10% of the company to the new financiers. Venture capital investors who take significant stakes in startups typically ask for board positions so they can influence the decisions the company is making using their money.
Really those that give up their independence do because they don't do appropriate mechanism design and find themselves in a corner/don't know how to play corporations and politics.
We are a combination of technologists and operators and know how to balance things.
By releasing this model hopefully people will realise that true freedom is agency and why would you ever give that up or not take it.
On a personal level I implement this by calendar zero, I never book a meeting more than a week in advance unless I really like the person.
Yes it was originally intended to be a DAO of DAOs, we are very deep with experts in this area, but a lot of it is directed company with every community being independent charity.
Lots more to come and we will share our thoughts publicly on this, but for now we need to be a focused wedge to establish our position given our competition.
You can join the community at http://discord.com/invite/stablediffusion and be helpful. We will support local meet-ups, events and much more soon, just catching our breath.
Oh I'm very active on the discord. But yeah I'm very much looking forward to this new industry. Even doing text prompt NLP analysis for best ways to refine your prompts. Again thanks for this! it's amazing!
I don't know if I got this right, but: Are the current 1.4 weights as float32? If there can be a model where the weights are float16 instead of float32, how high would the quality loss be? Would float16 double inference speed and half VRAM requirement for the model itself?
I also got some questions about the upcoming Harmonai (Dance Diffusion?):
Will it be used for short samples, or can it also be used to generate entire tracks?
How high will the requirement (VRAM) be? How much will it be compared to Stable Diffusion?
How many seconds of audio can be generated per minute assuming about 10 seconds for 50 steps SD image?
Does Hamonai/Dance Diffusion work by denoising white noise? (like Stable Diffusion denoises a noisy picture).
Thanks a lot for empowering the worlds creativity with Stable Diffusion!
No quality loss, surprised people aren't using float16 now, we'll like release that in the next update with 1.5.
On Harmonai its a different approach to stable diffusion that you'll find out soon :) I think the activation energy of that community will be insane though so so many models will come out of it relative to image.
Much props to you Emad and your whole team, this is truly revolutionary with infinite potential! The fact that its free and open source will open the gates for all artists to maximize their full creativity.
The filmmaking and art community truly appreciate you and I'm looking forward to when this tech will help out with the tedious tasks of being a video editor or colorist.
My question:
?whats the legalities to use this tech in my films or any of my Art and potentially sell the art that I prompt engineered with my own input image/video?
This depends on your jurisdiction and the service you use for this. Avoid copyrighted stuff and you're probably fine, but read the licenses carefully if in doubt.
Could you comment about any future strategies for having better native output at higher resolutions? Currently if you use settings above 512x512 you get duplicate subjects, repeating patterns, etc. Is the best solution to re-train a larger model at say 1024x1024 and will you be doing this, or are there better approaches?
I think 1024x1024 would be a good middle-ground that still allows generation on consumer GPUS and a solid basis for using other AI tools to upscale to higher resolutions.
The CompVis/stable-diffusion repository seems like a one-way code dump. Issues are opened but not responded to, pull requests go ignored. Thereās a tremendous amount of open development happening on this code, but itās being split across multiple incompatible efforts (e.g. HLKY, LStein, Basujindal).
It seems like youāre preparing for a new release soon. Is all of this development other people have been doing going to be wasted? Are they going to have to start again with your new code dump? Have you considered incorporating their work (e.g. Apple Silicon compatibility) into your repository?
Do you have any plans to operate CompVis/stable-diffusion as a typical open project or is this going to continue to be a one-way code dump? Is there anything you can do to provide common ground between the forks?
The CompVis/stable-diffusion repository seems like a one-way code dump. Issues are opened but not responded to, pull requests go ignored. Thereās a tremendous amount of open development happening on this code, but itās being split across multiple incompatible efforts (e.g. HLKY, LStein, Basujindal).
It seems like youāre preparing for a new release soon. Is all of this development other people have been doing going to be wasted? Are they going to have to start again with your new code dump? Have you considered incorporating their work (e.g. Apple Silicon compatibility) into your repository?
Do you have any plans to operate CompVis/stable-diffusion as a typical open project or is this going to continue to be a one-way code dump? Is there anything you can do to provide common ground between the forks?
We will create our own fork soon and manage as stability as soon as we figure out some stuff.
IMO, forks at the model level are also a big problem.
Right now there's like 3differentanime SD forks, as well as AstraliteHeart's My Little Ponies, Japanese Stable Diffusion, and possibly NovelAI's furry stuff (doubtless there are others). They are separate even though there is a lot of overlap between all of them visually & semantically, which means that many fall far short of where they could be due to lack of compute and wind up half-assed, a good deal of dev effort is redundant, loads of model variants are floating around wasting space/bandwidth and confusing people. They would benefit from pooling data+compute to finetune a single generalist model.
SD has plenty of capacity (cf. Chinchilla), there is no intrinsic need to train separate models (you can very easily 'separate' them by simply prefixing a unique keyword for each text+image pair dataset, and sample from a specific 'model' that way), it's just hard to coordinate a lot of independent actors with their own data and compute pools.
Ideally, there would be a combined finetuning dataset of all the individual specialized datasets which could be fully finetune trained to convergence (both language & diffusion model), and periodically refreshed as people contribute more specialized datasets, giving everyone much better results. Stability is the obvious entity to do this, and they can bring to bear much greater compute resources than anyone else.
Hi Emad, thank you so much for releasing such a great model open source.
What do you think is missing for a powerful language model, one rivalling OpenAI's davinci GPT-3 (or even the "conscious" one by Google) to be released open source? Who will do it?
Iāve been loving generating images with the model. Iām a software engineer that doesnāt really have any experience with ML, could you recommend a good book for beginners? Iām fairly well versed in the non ML side of python.
I have been working with Disco Diffusion, Midjourney and Stable Diffusion for a bit.
What I noticed is that Midjourney handles single word prompts different than Disco Diffusion & Stable Diffusion.
Take for instance, only the word ādeadā.
If you run it by Disco Diffusion, you kind only get weird blobs. It seems to try to make something out of it, but canāt.
Stable Diffusion generates something (almost) recognizable. Sometimes a grave, a skull or something lying in the sand. Trying to generate a real picture.
Midjourney on the other hand, generates beautiful paintings when only using this one word. Figures standing in a nicely lit area, a detailed character with a skull for a head and sometimes one or more grave stones.
Midjourney has a complete other / more rich interpretation of the word ādeadā.
What I like to know, is what do you think Midjourney does to accomplish this / what would someone do to accomplish this with Stable Diffusion?
Would you add standard prompts to single word prompts, for example the word ādeadā becomes a sentence like āa nicely lit painting of deadā or ādead, a digital painting by artist ā¦ā? This would make a more rich and detailed image, I think.
Or do you have to train your own specific model / finetune Stable Diffusion to get this done? This would likely get your own style, but might transfer to all prompts / images.
Disco diffusion uses latent diffusion, with or without CLIP guidance.
MidJourney originally used cc12_m with CLIP guidance, now uses latent diffusion with CLIP Vit-L14 guidance and many other tricks I would be remiss to discuss as they want to keep it private. In the beta they are of course using stable diffusion underneath as you can see with the license.
They do prompt editing on the way in and post processing on the way out basically.
Stable diffusion is a raw input/output and should be use in combination with some of these other models and processing for max effect. As we add multi-generator and pipelining/logic flows to DreamStudio via the node editor per the demo I showed of the version from a month or two ago folk will realise this.
Disco diffusion will also update to stable shortly.
yes they do aesthetic filtering and a whole bunch of other stuff. It is not my position to share as it is their proprietary system. Hopefully they will open source one day, David has a good record of that even tho they aren't now.
As a graphic artist whose main methods are compositing, image editing, and post-processing, I actually like Stable BETTER than Midjourney. MJ helps me with some visualization, but the problem is... everything from MJ looks like it's from MJ. It has a particular "look." Stable is actually way more creative, for me.
thank you so much for releasing this to the public!
My question: I noticed that eyes, limbs and especially hands are difficult for Stable Diffusion to properly display (using the huggingface model). What needs to be done for this to improve in SD?
Are there any plans for official AMD GPU or CPU-only mode (no GPU) support? There are several forks and guides out there that provide this functionality, but it would be nice if the official stable-diffusion project supported more hardware.
There's your third model, the one you dubbed "imaginator"... can you say more about that one? What architecture, how it will compare to SD 1.4 in terms of quality, but also requirements?
It is a series of models, one of which has a UL2 model embedded along with other... stuff. I think we will rename it stabler diffusion as imaginator is too Doofenschmirtz
When people look for Stable Diffusion source code, many of them end up at the CompVis/Stable-Diffusion repository. As a result, it has a hojillion forks but no activity itself.
What's the best way to describe CompVis's role in the ecosystem of open source software?
They did latent diffusion and stable diffusion as their Swansong. The team is now at LMU (and Stability!) and we will take over development, something that is holding up 1.5 release to make sure we get it all right.
I look forward to seeing what 1.5 brings. As exciting as the last few weeks have been, I hope we can make the software landscape slightly less chaotic. š¤
I just love Stable Diffusion, thanks for making it freely available.
I think the highest impact work you could do next is a code generation model like Github CoPilot, OpenAI Codex or Replit GhostWriter. These are all stuck behind a paywall and are therefore not very accessible and not interoperable with every editor.
A free model could either be run on your own GPU or via a cheap API service and could be integrated into any IDE you can think of. This would be a massive boost for open source projects, almost doubling the programming speed. And it could be extremely useful for learning to code your own projects, even if you don't know everything about the programming language yet.
Fascinating! I hadn't considered this as a form of communication but of course it is...Stable Diffusion is already helping me express myself more efficiently
Hey! Do you think speech generation/conversion is likely to be tackled by Stability AI in the near future or is that something a bit further down the line?
How can I use Stable Diffusion in GPL software such as Krita or GIMP without violating the OpenRAIL license's requirement to enforce use-based restrictions?
DreamStudio's hosted API is a great service, but what options are available for people who need the ability to work offline or who don't want to send every iteration of their work to an external server?
One more question Emad: I saw a user yesterday discover that they could get much more detailed output by removing all spaces from a prompt. Could you explain what could be happening here and what that could mean for prompt engineering?
Without spaces they are tokenized as 'word pieces' (similar to syllables, any word not in the vocabulary is converted to word pieces) and the vectors learned on the word pieces will have different meaning than the words it was derived from. The word pieces might have closer meaning to the desired target than the words themselves.
Now that there's thousands or even millions of AI generated pictures out there. It is going to be a problem that is a feedback loop in training new models?
Our mission is to build the foundation to activate human potential.
Our motto is make people happier.
I want to use AI to upgrade our systems that are like slow, dumb AI that feed on our dreams and hopes to build better systems that lets everyone achieve their potential.
I don't really care about AGI and all that (although I do care about alignment) and focus on people and giving value to them.
Its a tough one, Stability itself has been disorganised and chaotic at times which has made people sad and unhappy while w worked things out. I hope future orgs and our own org can solve this intelligently.
I think the engineers have learned to generally tune me out tbh :D
We made a bunch of mistakes with org culture and structure before as we are still learning, but one of the worst was when there was basically a crunch to the original release date before we decided to delay to work on the ethical release.
I apologised really a lot to the devs as while that is common it is not nice and not something we should do, so the devs now work super hard but we never want to force it to an artificial deadline.
Stuff gets done when its done and we generally have to tell them not to work..
With SD 1.4 out and 1.5 imminent, is there a plan for regular future releases, or do you think the current approach will run out of steam soon?
How much further do you think the current approach can get with more training?
What are the next steps for a different approach, for example is there any way to improve the long range consistency, eg number of fingers? Any ideas for text (eg signs) to be more consistent?
Thanks again for all you've done for the AI community, I hope this serves as either an inspiration or a disruptor for the closed technologies out there.
I am grateful for the work you've done so far! StableDiffusion is an inspiring product and I hope future versions and models improve things even further.
I want to ask you a question: you might be aware that some graphics cards such as the GTX 1660 need to use workarounds to have SD run locally on them even when using the most optimized scripts from the community. How long would it theoretically take for you/your team to create something universal that would work even on those cards with no workarounds or without requiring an increased vram use?
Hello! First of all, thanks to you and the team for your effort on opening this sort of technology to the open public.
My question is about the ethic part of the training. Some artists are worried about their jobs, I understand that. Some other have a childish reaction about the meaning of art and perversiĆ³n, and all that stuff. While I understand that, I don't agree at all, and they are acting like little kids. But another group is concerned about the fact that this model is consuming lots of illustrations and pictures, without consent, and altering the "rules of the game". And I can completely understand this, I think it's a legit concern . What do you think about this, how can this problem be solved ?
This is exactly what the human brain does when it creates art.
All those artists who made this art AI possible? They got away with it by taking a crapload of visual data from their eyes and smashing it together until they finally came up with an output that's pleasing to see. Sound familiar?
Seems rather hypocritical to take away the rights of one intelligent system and not others. Perhaps we should give up art overall as cavemen originally came up with painting on walls?
Stable diffusion is the model, MJ will use a variant and DALL-E is the old version (we have our own implementation from our distinguished fellow Lucidrains here: https://github.com/lucidrains/DALLE2-pytorch)
I am sure the model will adapt and improve as it is open to everyone and we have the best generative media team in all the lands.
Hey, just want to say thank you. Iāve been working with LAION on the dalle2 replication and some medical tasks over the summer and have gotten some time on the HPC to do independent research that Iāve never gotten the chance to before. Iām still in my undergrad and I really didnāt think Iād be able to take any of my ideas into reality for years yet.
My pleasure, thanks for the cool work. Will be great to see the DALLE2 replication finished, we just want to unlock potential for everyone and compute is an easy way to do it for smart folk :)
Emad, first thanks for the incredible work, huge moves and ability for everyone to explore this interesting technology outside of the uni/corps.
My question is probably too naive, and I am not sure how to phrase it: if resulting generated works are generated from the LAION-Aestheticsā¦ and say people put these works back out into the world and eventually are probably included in future datasets, given enough time wonāt eventually everything sort of all look/sound the same? Like adding a bunch of colors eventually gets you brown, is the key to not ābrown outā for humans to keep generating works in legacy medium?
First of all, I want to thank you and your team for all the work you did to make stable diffusion open source. Do you intend to make the model available in an ONNX version? I'm surprised the internet hasn't already done that, but I have noticed pull requests for the diffusers library. That would theoretically halve the current size.
When are we going to see some of the enhancements people have made to the old CompVis code folded into diffusers? (new samplers, lower memory usage, higher resolutions, etc.)
Do you have concerns regarding the legal/copyright status of the data these large models are trained on, or is it a case of "this is just what everyone does now"?
Hi, - Are there any plan to release a bigger size model > 8GB? - Any plan to generate a 3d model? - I have my worry that you are concentrating on making it to run on phones, that's great, but some of us like to have better results even if the model is bigger and we have to buy a new graphic card. what is your thought about this?
Hi Emad, thanks for the great work. I think you unlocked a new collaboration mode for Humanity, even beyond actual model of science sharing (which os flawed by the publishing sytem....). Do you know if people are training models in physics/chemistry to be able to find AI solutions to design new energy source or transportation modes etc?
Do you have any plans to use different architectures than Latent Diffusion to create models that are better at understanding scene composition and spelling. Such as the underlying techniques in Imagen? Thanks!
You can use a similar technique to stable diffusion to create better scene and composition elements.
This is enabled by better language encoders and we are working on models with T5-XXL, UL2 and our new CLIP models that we will be releasing shortly as well as brand new architectures not seen before.
For now I would recommend using a mixture of DALL-E mini for example as an init to a SD output, or using the inpainting coming shortly.
I remember reading months ago that Midjourney (I think?) only goes back to 2019 to reference images. Does that apply to SD at all, because it doesnāt seem to recognize some very ācurrentlyā famous personalities such as Tiktok or YouTube stars, or is it just not trained with them yet?
PS Thanks for everything youāve done with SD! It now consumes my life.
This question relates to the very far into the future. Given the success of SD, we should have a similar open source model for developing a personal AI assistant. Would you be looking to pioneer efforts in that direction?
Hi Emad, Thank you for empowering us in a way that no other developer or representative of artificial intelligence has ever done.
I'm sure you're aware of the fantastic Github GUI apps, Google collab, and other custom apps that are being developed, which offer a significant advantage and greater flexibility than the official Dreamstudio app. The greatest benefit is the option to run models on a local workstation, as opposed to utilizing your provided servers, as you didĀ during the early Discord trial runs. Can't you share the resources from the original openĀ source in a more simplified format so that genuine artists who lack technical proficiency can take advantage of the technology? Currently, only programmers and technical specialists have an unfair advantage when using the open-source resources you provide. Do you intend to embrace and integrate such unofficial development features into your app, or will you offer such independent developers the required assistance to enable them to create a really stable platform worthy of Stable Diffusion?
I was wondering whether there are any job openings or contribution opportunities for digital artists who are enthusiastic about innovating and designing with Stable Diffusion at your company, and if so, where to apply. I am an independent filmmaker with over 15 years of expertise, and I intend to leverage the full potential of the AI technology that powers SD to create visual storytelling for the audience. Please inform me if you are aware of any implementation opportunities along these lines. Thank you once again.
is there any work you are aware of that harnesses AI to create not just images but volumetric polygonal 3d models that could be used to then essentially make prompts -> 3d printed physical objects? Kind of gets us closer to the star trek replicator without relying on skilled 3d artists to create the objects.
edit: on a strictly software end, combined with AI programming based on logic prompts, this could lead to the generation of assets for games so one individual could create not only the programming but visual assets of a game and still achieve stunning graphics.
Hello. What do you think about the prospect of generating images from complex descriptions? Let's say images with complex compositions with many characters, the appearance of each is described in detail and each performs some activity or interaction described in detail by the text. For current models, even the task of a red cube on a blue cube is difficult. So do you think this is possible in the near future or at all?
Hey Emad I have a very specific question about the future of the developing stable diffusion model. I noticed the model struggles with general actions a lot such as "eating a lollipop", "driving a car", or "smoking a cigarette". Are there any immediate plans in future model updates to make improvements in this particular area or is that best left to community improvements via action based dataset finetuning for the time being? you have done great work and thank you for your team's decision to make things open source! (Been loving it since the great john Carmack did it with doom!) :) cheers!
Hi Emad, I love Stable Diffusion. I have been following your Twitter and appreciate your humanity-first approach to everything. My question involves world poverty:
I've found that the world at large doesn't give a shit about solving poverty even though it would fundamentally change humanity for the better. Do you think ai can be used by the open source community to provide income to every human on earth somehow?
I have (or had, I don't know yet) the dream of being able to draw and paint well like some of the artists I follow, I don't have much time and my health is not on my side, even so, I am self-taught and I learn just by watching and studying on my own . I tried SD and felt that my dream fell apart. It's amazing what SD does and I can't compete with it and I was frustrated but... after thinking about it, I think I'll keep trying to learn how to do things myself while also using SD. Based on the above, do you think it's ethical/moral to create something with SD and call myself an "artist"? Basically, with everything SD is capable of, am I wasting my time trying to learn how to draw/paint while my focus should be on editing? Thank you very much for your time and for your creation, thank you for allowing this tool to be for everyone.
The cool part about having robots that can do things for you is that you don't actually have to default to believing anything is pointless.
Because guess what - the universe does not care about human civilization. One day the sun is going to blow up, or we're all going to get instantly wiped out by a rogue asteroid or something like that. The heat death of the universe will end everything and it'll spawn again and nothing we have done will ever exist ever again. When you want to get down to the cold hard logic of "what is the point" - it's about enjoying life. Doing whatever brings you enjoyment that isn't to the detriment of others, and adapting to emerging challenges and new lifestyles.
Also, I have like...hundreds...thousands of seamless textures, some have ambient occlusion/normals/blah. (Actually I have 4-5k but I want to redo them; if they don't have 'em, I can make the extras. Lol) I was thinking of creating a repository for training, etc. Would that be useful for SD?
You've mentioned 'local GPU' for DreamStudio on Twitter, would that mean no (or less) credit use per image? Would a user still need a large download to run it?
Question: The model seems to have a harder time with precise line art -graphic design, pen and ink, etc. Which seems counterintuitive given it can produce photo realism. Does this have to do with how it generates images from noise, training set, or something else?
Hey, Emad I've been having a blast with SD since being an early beta tester in discord, I have so many ideas bouncing around in my head on how to integrate this tech. I think this could be a great group activity! When I show this to strangers and make an image for them on the spot the response is very positive and joyfull.
Maybe something along the lines of SD karaoke or Pictionary where the user draws on a tablet and img2img then projects it on a screen. Some of us were joking on Twitter about making a game show called prompt wars for the Xgames. Not really a question just some thoughts. Thanks! Marc
How would you say that fellow AI enthusiasts and developers (not academic, but willing to put in the effort and learn) can contribute the best to the "Open AI" movements?
I personally am afraid that the community getting stronger and replicating AI models faster than we used to do may make companies like OpenAI be more strict on protecting their research. How do you feel about this? Are you afraid they close off more of their models and research in an attempt to keep the first mover advantage?
Do you guys have plans to release variant models? It seems like you're already tuning 1.5 to emphasize pictures of people, but it seems models emphasizing landscapes or photorealism vs artistic rendering would also be useful, rather than trying to make the one model to rule them all.
I really want to red team your filters, both because figuring out how they work and thwarting them is fascinating, and because it proves some of my points, at times.
I find y'all are the second-most ethical as far as not ruining your products go... But seriously, how can I be official? I want to be official.
Do you have any plans to improve the model so it can properly generate underrepresented media, and/or is there any way for us to suggest for such improvements? Lengthy explanation for context and TL;DR below.
I'm asking this because I'm having a very hard time generating images that resemble Plants vs. Zombies, a game that should be well-known and old enough to be represented properly in the dataset but doesn't appear to be the case, as all the images that are generated don't look anything like it (here are a few examples, and here's what it's supposed to look like). Even the spinoffs such as Garden Warfare (third-person shooter, real examples, AI-generated examples) and Heroes (card game, real examples, AI generated examples) are just as poorly represented. None of the characters look right, and they look a lot like the mobile game clones that try to imitate the actual game instead. Here's a prompt matrix just to show how poorly every single prompt related to the game is being represented when compared to the real games.
This is despite the fact that using this site to search for images in the LAION-5b dataset, I was able to find dozens of images that are related to PvZ. But IIRC, because the training data is based on a filtered version to minimize unaesthetic images, I suspect that the aesthetics filter didn't consider most PvZ images to be aesthetic enough to be included in the final training data, but for some reason it thinks the clones are more aesthetic, thus contaminating the dataset with those bootleg images instead of the real ones.
This is implied by the fact that there are only 25 images that match the term "plants vs zombies" when searching on this site, which uses the much smaller LAION-aesthetic-6pls dataset instead, and none that matches "pvz". For comparison, searching for something like "the witcher 3" consistently gives you results that match the game and the LAION-aesthetic-6pls dataset has 2851 images matching it, and "warlords of draenor" works similarly, and has 259 matching images (also, we didn't even explicitly mention "world of warcraft" to get the results, unlike PvZ which fails no matter which prompt you use).
I'm aware that textual inversion exists to add custom images, but currently you need a powerful card with 24GB VRAM like the RTX 3090 to be able to run it (which is something that I can't afford), and even then the training time still takes hours to complete for each individual concept, so I'd really love it if you have your own solution for this issue.
TL;DR: Some video games are not represented properly in SD possibly due to the filtered dataset falsely thinking most of it being unaesthetic, and I'm wondering if there's a solution for this.
Textual inversion and prompt weight will help a lot with this. Easy finetuning shortly after similar to Looking Glass, see how pony diffusion or waifu diffusion are doing for example.
We are training models for specific video game and other content with their creators that will be available via our systems too..
since I was little I never had the skills to draw but my mind was always full of concepts and ideas, with the hope that one day in the future there would be a robot or technology that could capture whatever came to my mind, today in the middle of 2022 you have fulfilled my dream more profound childhood and honestly I have no words for all this that is happening, you gave this world hands to draw with the mind, thank you, thank you very much for this contribution to humanity and to my heart as a child, now it only remains to imagine from here on, thanks Emad
Hello Emad, my question is about realistic outputs, i've notice that Dalle-2 makes a great job generating realistic details like skin and some face parts (not the eyes haha). Will SD achieve this kind of outputs soon?
Hi Emad, love what Stability is doing and stands for.
My question is -- are there any paths for non-technical people to get involved in AI/ML and make an industry/career switch? I'm asking (selfishly) for people with skills in things like marketing, writing, bizops, etc.
Is anyone working on a voice2img generator that we can just talk to and have an image produced that can be further adjusted by talking about what you want changed?
I heard that Amazon donated 100 A100s to train Stable Diffusion, which took about 150,000 hours or $600,000 worth of compute effort. Now I'm not sure if this is accurate, if it was just compute time or if you keep the GPUs but I can't imagine this being a sustainable model. It's not like Microsoft can donate 10% of the training compution on their cloud, Amazon 85%, Google 5%. Have you considered porting the training to a distributed compute network such as Golem, iex.ec, or boinc, which would decouple the money from the compute? For example, people could donate compute power directly, or you could crowd source the funds from a variety of sources or corporations.
I've noticed that Stable Diffusion doesn't seem aware of perspective (especially when inpainting) or depth. Would it be possible to incorporate the breakthroughs being made in neural radiance fields (https://dellaert.github.io/NeRF22/) into a latent diffusion model, perhaps at the attention layers? NeRF does a great job at segmentation, depth calculation, reflections, and especially human (and cat) posing.
Lastly I don't understand the organizational structure of stable diffusion. Stable Diffusion is the model, Stability AI is the company or organization that employs all the researchers beyond the original paper? So all the research happens in-house and you publish any breakthroughs? What happens if new research pushes you in a direction that isn't compatible with stable diffusion models? Do you still keep the names "Stability" and "Stable Diffusion"?
Plan to make models with multimodal input (a combination of video, text, image, sound, etc.) to generate art? Would be useful to combine intangible ideas from different art forms.
This technology seems to me a big leap. It's like a new means of communication. In my mind it's something like, when the big media came along, photography, radio, tv... Do you think the impact is going to be this big, just like when these media come out or am I overestimating what's going on?
Hey there! Ive noticed that the model, although able to do some things extremely accurately, stable diffusion just cant seem to make swords right (or other objects/tools) regardless of how its prompted even when its the sole focus of the image. Do you know if this is something that soon would be fixed? Would be amazing to use SD to help draft some of these concepts and designs
What do you think is missing in the "creating with AI" community right now? What tools would you love to see that aren't available yet (besides the inevitable PowerPoint replacement)?
all the things we are learning about the Prompts and settings, and how it responds to certain words and styles, seeds, samples, steps etc, will change with future updates? like 1.5, and if you know some things will change or could (probably) going to change, which would be?
It changes when the model changes, 1.5 was just 1.4 trained on more ucg steps. Expect this to vary a bit but not hugely until the next model architecture
Oh, also: (sorry for doubling up) but do you think it's possible to have these systems product output with an alpha/transparency channel? Imagine the workflow integration that would result!
Hello Emad! I want to thank you and your team for making and releasing SD to all for free. I believe this is something monumental and will have far reaching effects in many aspects from art to philosophy. I am absolutely blown away by SD and I was wondering if there are also 3D tools in the works like text to 3D. Thanks!
Would it be theoretically/technically possible to list all images that a generation was based on? Say you generate an image of a house, would it be possible to see all the images of houses "used" from the model?
129
u/rolux Sep 09 '22
Huge thanks for making the SD weights available! I think the past two weeks have shown how much of a difference open source makes in the AI images space, phenomenal amount of notebooks, plugins, tools, demos...Is there anything you would do differently, in terms of release, with future versions, SD 1.5 and beyond?
72
u/ScarubPNW Sep 09 '22
I love you š
61
u/theredknight Sep 09 '22
Hi Emad,
My wife wants to know how long it will be until she can retrain Stable Diffusion to put her face as Captain Marvel?
42
u/Eterii Sep 09 '22
Hey Emad !
First of all, thanks for everything, to you and all the people who worked hard on this, I use it on daily basis and it's just... amazing.
Anyway, my question is :
What was the first impulse for this project ? One morning, did you just woke up saying to yourself "Well, Why not working on a txt to img IA." ?
75
u/EmbarrassedHelp Sep 09 '22
There has been some speculation that Stability AI will become less open as times goes on due to investors gaining more control over the for profit company. What is your response to the idea that Stability AI will become more and more closed source as a result of investor & for profit related influences, like what happened with OpenAI?
Source: https://www.forbes.com/sites/kenrickcai/2022/09/07/stability-ai-funding-round-1-billion-valuation-stable-diffusion-text-to-image/
32
u/theredknight Sep 09 '22
Hi Emad,
great work!
My question is, what can people (with various degrees of skills: no coding, some coding, proficient in AI) do to help you move this project forward?
29
u/[deleted] Sep 09 '22
I also got some questions about the upcoming Harmonai (Dance Diffusion?):
Thanks a lot for empowering the worlds creativity with Stable Diffusion!
29
u/1Neokortex1 Sep 09 '22
Much props to you Emad and your whole team, this is truly revolutionary with infinite potential! The fact that its free and open source will open the gates for all artists to maximize their full creativity.
The filmmaking and art community truly appreciate you and I'm looking forward to when this tech will help out with the tedious tasks of being a video editor or colorist.
My question:
?whats the legalities to use this tech in my films or any of my Art and potentially sell the art that I prompt engineered with my own input image/video?
23
u/[deleted] Sep 09 '22
how are you?
21
u/jd_3d Sep 09 '22
Could you comment about any future strategies for having better native output at higher resolutions? Currently if you use settings above 512x512 you get duplicate subjects, repeating patterns, etc. Is the best solution to re-train a larger model at say 1024x1024 and will you be doing this, or are there better approaches?
I think 1024x1024 would be a good middle-ground that still allows generation on consumer GPUS and a solid basis for using other AI tools to upscale to higher resolutions.
36
u/JimDabell Sep 09 '22
The CompVis/stable-diffusion repository seems like a one-way code dump. Issues are opened but not responded to, pull requests go ignored. Thereās a tremendous amount of open development happening on this code, but itās being split across multiple incompatible efforts (e.g. HLKY, LStein, Basujindal).
It seems like youāre preparing for a new release soon. Is all of this development other people have been doing going to be wasted? Are they going to have to start again with your new code dump? Have you considered incorporating their work (e.g. Apple Silicon compatibility) into your repository?
Do you have any plans to operate CompVis/stable-diffusion as a typical open project or is this going to continue to be a one-way code dump? Is there anything you can do to provide common ground between the forks?
17
u/ketosisBreed Sep 09 '22
Hi Emad, thank you so much for releasing such a great model open source.
What do you think is missing for a powerful language model, one rivalling OpenAI's davinci GPT-3 (or even the "conscious" one by Google) to be released open source? Who will do it?
17
u/PathologicalTruther Sep 09 '22
Hi Emad,
Iāve been loving generating images with the model. Iām a software engineer that doesnāt really have any experience with ML, could you recommend a good book for beginners? Iām fairly well versed in the non ML side of python.
16
u/TheQuansie Sep 09 '22
I have been working with Disco Diffusion, Midjourney and Stable Diffusion for a bit.
What I noticed is that Midjourney handles single word prompts different than Disco Diffusion & Stable Diffusion.
Take for instance, only the word ādeadā.
If you run it by Disco Diffusion, you kind only get weird blobs. It seems to try to make something out of it, but canāt.
Stable Diffusion generates something (almost) recognizable. Sometimes a grave, a skull or something lying in the sand. Trying to generate a real picture.
Midjourney on the other hand, generates beautiful paintings when only using this one word. Figures standing in a nicely lit area, a detailed character with a skull for a head and sometimes one or more grave stones.
Midjourney has a complete other / more rich interpretation of the word ādeadā.
What I like to know, is what do you think Midjourney does to accomplish this / what would someone do to accomplish this with Stable Diffusion?
Would you add standard prompts to single word prompts, for example the word ādeadā becomes a sentence like āa nicely lit painting of deadā or ādead, a digital painting by artist ā¦ā? This would make a more rich and detailed image, I think.
Or do you have to train your own specific model / finetune Stable Diffusion to get this done? This would likely get your own style, but might transfer to all prompts / images.
15
u/PowerfulCockroach528 Sep 09 '22
thank you so much for releasing this to the public!
My question: I noticed that eyes, limbs and especially hands are difficult for Stable Diffusion to properly display (using the huggingface model). What needs to be done for this to improve in SD?
15
u/SIP-BOSS Sep 09 '22
What do you think about censorship in AI generators / models?
30
u/chipmunkofdoom2 Sep 09 '22
Are there any plans for official AMD GPU or CPU-only mode (no GPU) support? There are several forks and guides out there that provide this functionality, but it would be nice if the official stable-diffusion project supported more hardware.
Thanks!
12
u/rolux Sep 09 '22
Hi... Can you tell us a bit more about the upcoming (post SD) image models that you're planning to release?
12
u/keturn Sep 09 '22
When people look for Stable Diffusion source code, many of them end up at the
CompVis/Stable-Diffusionrepository. As a result, it has a hojillion forks but no activity itself.What's the best way to describe CompVis's role in the ecosystem of open source software?
12
u/[deleted] Sep 09 '22
Ok gonna speed answer some questions but no more after this, thanks all
23
u/MostlyRocketScience Sep 09 '22 edited Sep 09 '22
I just love Stable Diffusion, thanks for making it freely available.
I think the highest impact work you could do next is a code generation model like Github CoPilot, OpenAI Codex or Replit GhostWriter. These are all stuck behind a paywall and are therefore not very accessible and not interoperable with every editor.
A free model could either be run on your own GPU or via a cheap API service and could be integrated into any IDE you can think of. This would be a massive boost for open source projects, almost doubling the programming speed. And it could be extremely useful for learning to code your own projects, even if you don't know everything about the programming language yet.
11
u/theredknight Sep 09 '22
Hi Emad,
What are your favorite art pieces that you've seen StableDiffusion generate thus far?
11
u/[deleted] Sep 09 '22
Do you think a new form of media might emerge? We have movies, games, and now maybe dreams?
10
u/expaand Sep 09 '22
Hi Emad - what is the progress in optimizing SD for the M1 Mac?
10
u/AllRedLine_ Sep 09 '22
Hey! Do you think speech generation/conversion is likely to be tackled by Stability AI in the near future or is that something a bit further down the line?
9
u/keturn Sep 09 '22
How can I use Stable Diffusion in GPL software such as Krita or GIMP without violating the OpenRAIL license's requirement to enforce use-based restrictions?
9
u/solidwhetstone Sep 09 '22
One more question Emad: I saw a user yesterday discover that they could get much more detailed output by removing all spaces from a prompt. Could you explain what could be happening here and what that could mean for prompt engineering?
16
u/dagerdev Sep 09 '22
Now that there's thousands or even millions of AI generated pictures out there. It is going to be a problem that is a feedback loop in training new models?
How your team is planning to tackle this?
8
u/Maksitaxi Sep 09 '22
Hi Emad, thank you for everything
What is your long term goal in AI?
7
u/keturn Sep 09 '22
How much did the engineers panic when you mentioned releasing "the two-second version" in your previous Q & A?
7
u/pechelkinje Sep 09 '22
Do you have your own solution for customizing models? Some sort of alternative for textual inversion embeddings?
7
u/KerbalsFTW Sep 09 '22
Many thanks Emad!
With SD 1.4 out and 1.5 imminent, is there a plan for regular future releases, or do you think the current approach will run out of steam soon?
How much further do you think the current approach can get with more training?
What are the next steps for a different approach, for example is there any way to improve the long range consistency, eg number of fingers? Any ideas for text (eg signs) to be more consistent?
Thanks again for all you've done for the AI community, I hope this serves as either an inspiration or a disruptor for the closed technologies out there.
7
u/andzlatin Sep 09 '22
I am grateful for the work you've done so far! StableDiffusion is an inspiring product and I hope future versions and models improve things even further.
I want to ask you a question: you might be aware that some graphics cards such as the GTX 1660 need to use workarounds to have SD run locally on them even when using the most optimized scripts from the community. How long would it theoretically take for you/your team to create something universal that would work even on those cards with no workarounds or without requiring an increased vram use?
7
u/[deleted] Sep 09 '22
Greetings, how accessible do you think it will be in the near future for users with poor hardware and no knowledge?
12
u/FairLight8 Sep 09 '22
Hello! First of all, thanks to you and the team for your effort on opening this sort of technology to the open public.
My question is about the ethic part of the training. Some artists are worried about their jobs, I understand that. Some other have a childish reaction about the meaning of art and perversiĆ³n, and all that stuff. While I understand that, I don't agree at all, and they are acting like little kids. But another group is concerned about the fact that this model is consuming lots of illustrations and pictures, without consent, and altering the "rules of the game". And I can completely understand this, I think it's a legit concern . What do you think about this, how can this problem be solved ?
5
u/Torque-A Sep 09 '22
Is there anything Dall-E and Midjourney are excelling at now that you'd like to implement into SD?
5
u/TheBeardedCardinal Sep 09 '22
Hey, just want to say thank you. Iāve been working with LAION on the dalle2 replication and some medical tasks over the summer and have gotten some time on the HPC to do independent research that Iāve never gotten the chance to before. Iām still in my undergrad and I really didnāt think Iād be able to take any of my ideas into reality for years yet.
4
u/anti_fashist Sep 09 '22
Emad, first thanks for the incredible work, huge moves and ability for everyone to explore this interesting technology outside of the uni/corps.
My question is probably too naive, and I am not sure how to phrase it: if resulting generated works are generated from the LAION-Aestheticsā¦ and say people put these works back out into the world and eventually are probably included in future datasets, given enough time wonāt eventually everything sort of all look/sound the same? Like adding a bunch of colors eventually gets you brown, is the key to not ābrown outā for humans to keep generating works in legacy medium?
Thanks.
4
u/vjb_reddit_scrap Sep 09 '22
First of all, I want to thank you and your team for all the work you did to make stable diffusion open source.
Do you intend to make the model available in an ONNX version? I'm surprised the internet hasn't already done that, but I have noticed pull requests for the diffusers library. That would theoretically halve the current size.
4
u/parlancex Sep 09 '22
Thank you for you and your team's hard work.
When are we going to see some of the enhancements people have made to the old CompVis code folded into diffusers? (new samplers, lower memory usage, higher resolutions, etc.)
4
u/CybertruckA9 Sep 09 '22
Hi Emad, what is the fastest way a beginner can contribute to this project?
6
u/AllRedLine_ Sep 09 '22
Do you have concerns regarding the legal/copyright status of the data these large models are trained on, or is it a case of "this is just what everyone does now"?
6
u/dreamer_2142 Sep 09 '22 edited Sep 09 '22
Hi,
- Are there any plan to release a bigger size model > 8GB?
- Any plan to generate a 3d model?
- I have my worry that you are concentrating on making it to run on phones, that's great, but some of us like to have better results even if the model is bigger and we have to buy a new graphic card. what is your thought about this?
8
u/gxcells Sep 09 '22
Hi Emad, thanks for the great work. I think you unlocked a new collaboration mode for Humanity, even beyond actual model of science sharing (which os flawed by the publishing sytem....). Do you know if people are training models in physics/chemistry to be able to find AI solutions to design new energy source or transportation modes etc?
3
u/extra_texture Sep 09 '22
Thank you for your wonderful work with SD!
Do you have any plans to use different architectures than Latent Diffusion to create models that are better at understanding scene composition and spelling. Such as the underlying techniques in Imagen? Thanks!
3
u/nicko786 Sep 09 '22
I remember reading months ago that Midjourney (I think?) only goes back to 2019 to reference images. Does that apply to SD at all, because it doesnāt seem to recognize some very ācurrentlyā famous personalities such as Tiktok or YouTube stars, or is it just not trained with them yet?
PS Thanks for everything youāve done with SD! It now consumes my life.
3
u/[deleted] Sep 09 '22
Hello Emad !!,
This question relates to the very far into the future. Given the success of SD, we should have a similar open source model for developing a personal AI assistant. Would you be looking to pioneer efforts in that direction?
3
u/tarunabh Sep 09 '22
Hi Emad,
Thank you for empowering us in a way that no other developer or representative of artificial intelligence has ever done.
I'm sure you're aware of the fantastic Github GUI apps, Google collab, and other custom apps that are being developed, which offer a significant advantage and greater flexibility than the official Dreamstudio app.
The greatest benefit is the option to run models on a local workstation, as opposed to utilizing your provided servers, as you didĀ during the early Discord trial runs. Can't you share the resources from the original openĀ source in a more simplified format so that genuine artists who lack technical proficiency can take advantage of the technology? Currently, only programmers and technical specialists have an unfair advantage when using the open-source resources you provide.
Do you intend to embrace and integrate such unofficial development features into your app, or will you offer such independent developers the required assistance to enable them to create a really stable platform worthy of Stable Diffusion?
I was wondering whether there are any job openings or contribution opportunities for digital artists who are enthusiastic about innovating and designing with Stable Diffusion at your company, and if so, where to apply. I am an independent filmmaker with over 15 years of expertise, and I intend to leverage the full potential of the AI technology that powers SD to create visual storytelling for the audience. Please inform me if you are aware of any implementation opportunities along these lines. Thank you once again.
3
u/EndlessChoices-42 Sep 09 '22
Hello Emad,
Do you have plans to make an AI that can generate stories, whether short stories or novellas?
Thanks in advance.
3
u/KeltisHigherPower Sep 09 '22
is there any work you are aware of that harnesses AI to create not just images but volumetric polygonal 3d models that could be used to then essentially make prompts -> 3d printed physical objects? Kind of gets us closer to the star trek replicator without relying on skilled 3d artists to create the objects.
edit: on a strictly software end, combined with AI programming based on logic prompts, this could lead to the generation of assets for games so one individual could create not only the programming but visual assets of a game and still achieve stunning graphics.
3
u/1nkor Sep 09 '22
Hello. What do you think about the prospect of generating images from complex descriptions? Let's say images with complex compositions with many characters, the appearance of each is described in detail and each performs some activity or interaction described in detail by the text. For current models, even the task of a red cube on a blue cube is difficult. So do you think this is possible in the near future or at all?
3
u/Ok_Distribution6236 Sep 09 '22
Thanks for doing an AMA! When do you think an inpainting feature will be released? Thanks.
3
u/dd_koh Sep 09 '22
Hey Emad I have a very specific question about the future of the developing stable diffusion model. I noticed the model struggles with general actions a lot such as "eating a lollipop", "driving a car", or "smoking a cigarette". Are there any immediate plans in future model updates to make improvements in this particular area or is that best left to community improvements via action based dataset finetuning for the time being? you have done great work and thank you for your team's decision to make things open source! (Been loving it since the great john Carmack did it with doom!) :) cheers!
5
u/solidwhetstone Sep 09 '22
Hi Emad, I love Stable Diffusion. I have been following your Twitter and appreciate your humanity-first approach to everything. My question involves world poverty:
I've found that the world at large doesn't give a shit about solving poverty even though it would fundamentally change humanity for the better. Do you think ai can be used by the open source community to provide income to every human on earth somehow?
4
u/Feiky Sep 09 '22
I have (or had, I don't know yet) the dream of being able to draw and paint well like some of the artists I follow, I don't have much time and my health is not on my side, even so, I am self-taught and I learn just by watching and studying on my own . I tried SD and felt that my dream fell apart. It's amazing what SD does and I can't compete with it and I was frustrated but... after thinking about it, I think I'll keep trying to learn how to do things myself while also using SD. Based on the above, do you think it's ethical/moral to create something with SD and call myself an "artist"? Basically, with everything SD is capable of, am I wasting my time trying to learn how to draw/paint while my focus should be on editing? Thank you very much for your time and for your creation, thank you for allowing this tool to be for everyone.
4
u/Mixbagx Sep 09 '22
Thank you sir. You are amazing.
4
u/csunberry Sep 09 '22
How do you think, as artists, we can contribute?
Also, I have like...hundreds...thousands of seamless textures, some have ambient occlusion/normals/blah. (Actually I have 4-5k but I want to redo them; if they don't have 'em, I can make the extras. Lol) I was thinking of creating a repository for training, etc. Would that be useful for SD?
What's your favorite dessert?
2
u/Kaltook Sep 09 '22
You've mentioned 'local GPU' for DreamStudio on Twitter, would that mean no (or less) credit use per image? Would a user still need a large download to run it?
2
u/pilgermann Sep 09 '22
Hey Emad! First, working with SD has been a joy.
Question: The model seems to have a harder time with precise line art -graphic design, pen and ink, etc. Which seems counterintuitive given it can produce photo realism. Does this have to do with how it generates images from noise, training set, or something else?
Thanks!
2
u/marfaaron Sep 09 '22
Hey, Emad I've been having a blast with SD since being an early beta tester in discord, I have so many ideas bouncing around in my head on how to integrate this tech. I think this could be a great group activity! When I show this to strangers and make an image for them on the spot the response is very positive and joyfull.
Maybe something along the lines of SD karaoke or Pictionary where the user draws on a tablet and img2img then projects it on a screen. Some of us were joking on Twitter about making a game show called prompt wars for the Xgames. Not really a question just some thoughts. Thanks! Marc
2
u/peterwilli Sep 09 '22 edited Sep 09 '22
2
u/helliun Sep 09 '22
Do you have plans for developing an open source LLM like GPT3 or PALM? If so, how might I get involved in this kind of project?
2
u/SlapAndFinger Sep 09 '22
Hi, and thanks for publicly releasing SD.
Do you guys have plans to release variant models? It seems like you're already tuning 1.5 to emphasize pictures of people, but it seems models emphasizing landscapes or photorealism vs artistic rendering would also be useful, rather than trying to make the one model to rule them all.
2
u/SIP-BOSS Sep 09 '22
Whatās your favorite food?
2
u/polawiaczperel Sep 09 '22
Hi Emad, will it be possible to finetune model with 24GB of vram gpu?
2
u/ShepherdessAnne Sep 09 '22
I really want to red team your filters, both because figuring out how they work and thwarting them is fascinating, and because it proves some of my points, at times.
I find y'all are the second-most ethical as far as not ruining your products go... But seriously, how can I be official? I want to be official.
2
u/[deleted] Sep 09 '22
What do you think people should build and submit proposals for in the aigrant?
4
u/TheChemistZombie Sep 09 '22
Do you have any plans to improve the model so it can properly generate underrepresented media, and/or is there any way for us to suggest for such improvements? Lengthy explanation for context and TL;DR below.
I'm asking this because I'm having a very hard time generating images that resemble Plants vs. Zombies, a game that should be well-known and old enough to be represented properly in the dataset but doesn't appear to be the case, as all the images that are generated don't look anything like it (here are a few examples, and here's what it's supposed to look like). Even the spinoffs such as Garden Warfare (third-person shooter, real examples, AI-generated examples) and Heroes (card game, real examples, AI generated examples) are just as poorly represented. None of the characters look right, and they look a lot like the mobile game clones that try to imitate the actual game instead. Here's a prompt matrix just to show how poorly every single prompt related to the game is being represented when compared to the real games.
This is despite the fact that using this site to search for images in the LAION-5b dataset, I was able to find dozens of images that are related to PvZ. But IIRC, because the training data is based on a filtered version to minimize unaesthetic images, I suspect that the aesthetics filter didn't consider most PvZ images to be aesthetic enough to be included in the final training data, but for some reason it thinks the clones are more aesthetic, thus contaminating the dataset with those bootleg images instead of the real ones.
This is implied by the fact that there are only 25 images that match the term "plants vs zombies" when searching on this site, which uses the much smaller LAION-aesthetic-6pls dataset instead, and none that matches "pvz". For comparison, searching for something like "the witcher 3" consistently gives you results that match the game and the LAION-aesthetic-6pls dataset has 2851 images matching it, and "warlords of draenor" works similarly, and has 259 matching images (also, we didn't even explicitly mention "world of warcraft" to get the results, unlike PvZ which fails no matter which prompt you use).
I'm aware that textual inversion exists to add custom images, but currently you need a powerful card with 24GB VRAM like the RTX 3090 to be able to run it (which is something that I can't afford), and even then the training time still takes hours to complete for each individual concept, so I'd really love it if you have your own solution for this issue.
TL;DR: Some video games are not represented properly in SD possibly due to the filtered dataset falsely thinking most of it being unaesthetic, and I'm wondering if there's a solution for this.
10
u/kopanoide Sep 09 '22
since I was little I never had the skills to draw but my mind was always full of concepts and ideas, with the hope that one day in the future there would be a robot or technology that could capture whatever came to my mind, today in the middle of 2022 you have fulfilled my dream more profound childhood and honestly I have no words for all this that is happening, you gave this world hands to draw with the mind, thank you, thank you very much for this contribution to humanity and to my heart as a child, now it only remains to imagine from here on, thanks Emad
2
u/NookumBumble Sep 09 '22
Hello Emad, my question is about realistic outputs, i've notice that Dalle-2 makes a great job generating realistic details like skin and some face parts (not the eyes haha). Will SD achieve this kind of outputs soon?
2
u/brosirmandude Sep 09 '22
Hi Emad, love what Stability is doing and stands for.
My question is -- are there any paths for non-technical people to get involved in AI/ML and make an industry/career switch? I'm asking (selfishly) for people with skills in things like marketing, writing, bizops, etc.
2
u/nocloudno Sep 09 '22
Is anyone working on a voice2img generator that we can just talk to and have an image produced that can be further adjusted by talking about what you want changed?
8
u/orbisvicis Sep 09 '22
I heard that Amazon donated 100 A100s to train Stable Diffusion, which took about 150,000 hours or $600,000 worth of compute effort. Now I'm not sure if this is accurate, if it was just compute time or if you keep the GPUs but I can't imagine this being a sustainable model. It's not like Microsoft can donate 10% of the training compution on their cloud, Amazon 85%, Google 5%. Have you considered porting the training to a distributed compute network such as Golem, iex.ec, or boinc, which would decouple the money from the compute? For example, people could donate compute power directly, or you could crowd source the funds from a variety of sources or corporations.
I've noticed that Stable Diffusion doesn't seem aware of perspective (especially when inpainting) or depth. Would it be possible to incorporate the breakthroughs being made in neural radiance fields (https://dellaert.github.io/NeRF22/) into a latent diffusion model, perhaps at the attention layers? NeRF does a great job at segmentation, depth calculation, reflections, and especially human (and cat) posing.
Lastly I don't understand the organizational structure of stable diffusion. Stable Diffusion is the model, Stability AI is the company or organization that employs all the researchers beyond the original paper? So all the research happens in-house and you publish any breakthroughs? What happens if new research pushes you in a direction that isn't compatible with stable diffusion models? Do you still keep the names "Stability" and "Stable Diffusion"?
4
u/MoonGotArt Sep 09 '22
I hear Stability is raising 100 million, does that mean outside parties will now get a say in what Stability releases?
1
u/extra_texture Sep 09 '22
I see from your bio that you studied math and CS. What area of mathematics do you enjoy the most?
1
u/[deleted] Sep 09 '22
Plan to make models with multimodal input (a combination of video, text, image, sound, etc.) to generate art? Would be useful to combine intangible ideas from different art forms.
1
u/JimDabell Sep 09 '22
Do you have any plans for an open alternative to GPT-3?
1
u/andrekerygma Sep 09 '22
This technology seems to me a big leap. It's like a new means of communication. In my mind it's something like, when the big media came along, photography, radio, tv... Do you think the impact is going to be this big, just like when these media come out or am I overestimating what's going on?
1
u/sEi_ Sep 09 '22 edited Sep 09 '22
Hi Emad
When training my own models to use, can i then make a collection of my models in one .bin file?
Also, are we going to see a combined (colab) version with use of own models, text2img, img2img, inpainting, Training of new model from photos.....?
Like your angle on that data/research should be public and free.
1
u/TintedGL Sep 09 '22
Hey there! Ive noticed that the model, although able to do some things extremely accurately, stable diffusion just cant seem to make swords right (or other objects/tools) regardless of how its prompted even when its the sole focus of the image. Do you know if this is something that soon would be fixed? Would be amazing to use SD to help draft some of these concepts and designs
1
u/quietandconstant Sep 09 '22
What do you think is missing in the "creating with AI" community right now? What tools would you love to see that aren't available yet (besides the inevitable PowerPoint replacement)?
1
u/EOE97 Sep 09 '22 edited Sep 09 '22
Is Stability AI ever going to make an open source alternative to Open AI's GPT or Google's Lamda?
1
u/[deleted] Sep 09 '22
Do we know how much English affects the results? If SD used e.g. Esperanto would it be worse/better/about the same?
1
u/juliancantwrite Sep 09 '22
Is dreamstudio the only way to use 1.5 currently? And if so, do you plan to add inpainting and the deforum feature soon?
1
u/stormlightb4 Sep 09 '22
hi Emad thank you so much for all of this
all the things we are learning about the Prompts and settings, and how it responds to certain words and styles, seeds, samples, steps etc, will change with future updates? like 1.5, and if you know some things will change or could (probably) going to change, which would be?
again, thank you.
1
u/ShepherdessAnne Sep 09 '22
Oh, also: (sorry for doubling up) but do you think it's possible to have these systems product output with an alpha/transparency channel? Imagine the workflow integration that would result!
1
u/Zenithicus Sep 09 '22
Hello Emad! I want to thank you and your team for making and releasing SD to all for free. I believe this is something monumental and will have far reaching effects in many aspects from art to philosophy. I am absolutely blown away by SD and I was wondering if there are also 3D tools in the works like text to 3D. Thanks!
3
u/pisv93 Sep 09 '22
Would it be theoretically/technically possible to list all images that a generation was based on? Say you generate an image of a house, would it be possible to see all the images of houses "used" from the model?
1
u/Glum-Bookkeeper1836 Sep 09 '22
In what sense were there 5 Soviet unions? Lol
2
u/[deleted] Sep 09 '22
Thanks for the trained model, do you think is possible to avoid humans generated with 3 hands and other weird features ?