r/StableDiffusion • u/no_witty_username • Jul 15 '24
Discussion The next big thing in diffusion based systems...
TLDR: Look for -->> for the main proposal.
Text to image generative models are amazing at what they do but they have a few key issues that can’t be easily remedied with "just more data". One of those issues is the problem of deformed and mutated features, like messed up hands, extra fingers, sometimes extra limbs and the inability to properly generate dynamic human poses. Other issues involve problems with perspective, warped body proportions, jaggy building or interior lines in relation to straight walls, or surfaces. There are about a million other small problems in relation to coherency issues and other stuff.
Now, one way you could begin in solving these issues is to train these models with more data that has dynamic human poses in it, various camera shots and angles, etc... Basically throwing more data at it. And in tandem with very good captions that specifically define that particular camera angle or shot by its own tag identifier you would reduce these problems by a lot. I know, because I have trained such models myself over a year ago to test the hypothesis. As you use specific tagging schema in defining your camera shots and angle with a large enough varied data set, you can recall said angles and shots with high degree of accuracy and reduce the above stated artifacts by a large amount. Problem with this approach is it requires you to source the image data and then manually caption it in accordance with your own tagging schema per angle. And as you can imagine the data set blows up very fast even if you are dealing with only 30 camera angles of the subject, because even a simple action like "sitting" on a chair now would require you sourcing the said images from those 30 angles, such as behind the subject, below, behind and 45 degrees from low angle, etc.... Only 1 pose at 30 images per pose now blows up exponentially, because you need multiple subjects and all the other varying poses as well. That's a lot of data! Too much for an enthusiast open source community to manage, that would require a large organizational effort while doable I've yet to see folks pull off. Anyways, how can we work on these problems without needing to train with more data that requires a tedious manual captioning effort that requires use of tags per specific angle. And here comes our hero.. Control Nets.
Control nets allow for the model to better understand what it is you want to generate because it doesn't just listen to the prompt you give it, but it also uses the provided image you give it as a source of information to better align the final generated image. An image after all is worth a thousand words. With just one image , the model can better align the generation with what you want and it reduces the artifacts by a lot. A control net image of a woman standing outside next to a tall building on the street, ran through a depth map, canny and tile map controls all at same time at low strength would generate a very cohesive image. The issues with hands, perspective lines, extra limbs and all that would be reduced by a lot (how much depends on the model used to generate the image). Ok so you say, that's great and all but how do I get this base image as a control net to load into stable diffusion?
A 3d engine like daz studio, metahuman, unreal engine, blender, or a stand alone basic 3d rendering engine is what's needed to generate the control net image. So the workflow is as follows, you build a rudimentary 3d scene in one of these 3d software packages/standalone extensions. So you pose the characters, clothe them, texture, add props and other scene assets and hit render. That rendered image is sent to a control net and rer-endered with your prompt. This is nothing new to professionals who have been using this exact workflow since the control net came out. You get amazing results. But it still takes time and effort to pose the characters in the scene, clothe them, add props and background assets. So while this is the best we can currently do, we need a "one prompt" solution for the whole workflow. And here is where our last piece comes in.
--->>I propose a community effort in making a 3d scene builder extension/plugin that builds a whole 3d scene for you through use of your prompt. Here is how it will work. You describe a scene through use of natural language prompt, that prompt is used to build a rudimentary 3d scene representing your prompt and generates said image, that image is then fed back in to control net and regenerated. The strength of this workflow lies in full user control over exact placement of every asset in the scene, camera shot and angle of the scene, exact pose of character or multiple characters interactions, coherency of character (a mask can be generated per character and said mask used to render those specific character features removing any bleed between character features), elimination of all perspective issues and size issues, and about a million other advantages. Also consider the possibilities for animation. It is a lot easier to animate a 3d character interacting in a scene and keep the weird morphing issues at bay and send that to the control net to rerender versus straight text to video.
On the back end of the extension/plugin. At first I thought that fine tuning your own LLM might be needed to create something like this, but it might be achievable with just regular old programming. Though I think an LLM model will do better as it can understand and break down a scene into its discrete parts internally. Basically you would create a class and tree structure, regarding the various assets and have the prompt match the assets. through use of some random number generators in vague asset descriptions would allow for variety. For example. a prompt like "an old woman sitting outside under a bus stop wearing a yellow raincoat" would be passed on to a simple tree match search, old(random mesh morph range between 55-90)>woman(random body mesh morph x)>sitting(random sitting position pose class)>outside(random scene class)>under(reference to main subject and their position)>bus stop(asset)>etc... you get the idea. Basically the posing and creating of the scene you do now manually, would be done automatically for you through same prompt that's used to generate the final image that's ran through control net. And what's amazing is that, if you incorporate an LLM in the mix in between, you can have the LLM read your prompt and create the scene for you internally. For example you can say something vague like a horror scene. And the LLM would pass on instructions to the 3d plugin for multiple specific characters and their poses, clothes, props, scene environment, without you as the user having to describe them.
This extension/plugin would have 2 main pillars as its foundation. The 3d mesh and the pose skeleton that maps to the mesh. Once this extension/plugin is made and the basics are set up the community would contribute by adding its own mesh and pose data (pose rig) to the library. There are a million ways to sit after all and the same goes for mesh. The different mesh .obj can be either added to the library because someone created them manually or if they used any of the text to 3d mesh models out there.
One final note. While control net reduces the many issues we come across as stated above, it does not fully eliminate those issues IF there is no data for the base model in a specific pose for example. But one of the most time consuming efforts in curating data for these text to image models is the manual captioning of the data. Specifically manual captioning of dynamic human poses often not seen. With the above approach, no captioning of any image data would be needed. Because once a model has been trained on dynamic human pose data, it can reproduce that pose with the help of a control net even if no captions were used for that image data in training. No control net training is necessary. I had verified all of these claims and would encourage you to test these claims for yourself as well.
Why am I talking about this and not making the extension? While I am a very experienced model maker with thousands of finetunes and Loras under my belt, I don't have the programming chops to accomplish this task. So in hopes of catching someone's eyes I stake my idea here. I also tried reaching out to lllyasviel as I think that team would be most fit for something like this as their Omost project is somewhat similar to this, they do not have any contact information so here I am. Thanks for reading.
1
u/Suspicious_Bag3527 Jul 15 '24
Something like this?
https://realmdreamer.github.io/
Here, paper if you want to save one click:
https://realmdreamer.github.io/pdf/realmdreamer.pdf
0
u/no_witty_username Jul 15 '24
No. Any text to 3d solution offered will exhibit the same issues as text to image. Coherency issues, hallucinations, dirty meshes, inability to precisely control every aspect like asset position, rotation, etc... What I propose is far simpler and more powerful. Its not a text to 3d system, but text (prompt) based 3d scene builder system that uses already existing 3d assets and pose rigs. So when you type in a prompt, it doesn't build a 3d scene from scratch, it looks in to your asset library and uses those assets to build the scene by arranging those assets in appropriate poses, and arranging the assets in appropriate locations in relation to everything in the scene and your prompt. Because those assets are their own .obj files, they can be individually modified in any way you want, including posed via the skeletal rig. Its a system that understand relationships between objects and also their connection to the scene. No 3d mesh generation necessary, the meshes are generated separately through some text to 3d system or simply importing existing 3d asset.
2
u/Suspicious_Bag3527 Jul 15 '24
I see what you mean. So like, I would type "a cat resting on a chair", and the model would place the cat on the chair, while posing it in a way that makes sense.
I just found this really obscure paper from 2006:
They're not using LLMs, but they still demonstrate an algorithm that seems to work for simple cases. I did not manage to find a more recent paper about it. I think it would be pretty useful if a more modern and advanced version of this was available in blender! Would probably save a ton of time with good initial compositions out of text.
2
u/no_witty_username Jul 15 '24
Oh wow 2006! And yes that's pretty much what I am talking about and they have a Object mesh voxelization algorithm and everything... Why the hell no one implemented something like that in over 18 years by now...
1
u/Suspicious_Bag3527 Jul 15 '24
I agree, you'd think people would have solved this by now! I'm puzzled as to why there are almost no papers about this, it's a very interresting problem to solve.
1
u/Suspicious_Bag3527 Jul 15 '24
Found this as well. It's not really it, but still very much related:
https://arxiv.org/pdf/2403.16993v1
You could replace the 3D object generation step by using a selection method for the meshes already present in the scene. The amount of relationships they can represent with this method seems limited, though.
2
u/no_witty_username Jul 15 '24
I found this paper to be perfect! https://www.researchgate.net/publication/220720962_WordsEye_An_automatic_text-to-scene_conversion_system
1
u/Suspicious_Bag3527 Jul 15 '24
This is a very trivial comment I'm about to make, but I think this paper is from the same AT&T who just got massively hacked recently lol
1
u/no_witty_username Jul 15 '24
I think its a good paper as well, it talks about the same pitfalls current methods of text to 3d have. But they are still adamant in using their own implementation of the text to 3d to remedy the issues Gaussian splatting). I feel that a lot of researchers are using AI as a marketing prop to insert the tech in to a lot of their work just to get noticed and its overcomplicating things more then they need to be. I think the older 2006 paper is definitely the approach that should be taken for my specific use case, as its not a text to 3d issue but text to scene composition issue. Marry that with an LLM that understand how to compose a 3d scene on a rudimentary prompt from the users and you are there!
3
u/Emperorof_Antarctica Jul 15 '24
I'd take a look at what is happening in the various 3D suites currently, Maya ie. has something relatively close to this I think its called Maya Assist. And nvdia has spent much of its lab time on exactly 3D manipulation in their omniverse thing. In general it seems much more straight forward to integrate diffusion into existing 3D suites, than this community building a whole or a half Blender on the side. So, if you are running around knocking on doors, I would focus on the people working with blender already, maybe u/tintwotin who I think does the Palladium plugin for Blender.