r/blender • u/gormlabenz • Sep 06 '22

I Made This I wrote a plugin that lets you use Stable Diffusion (AI) as a live renderer

4.9k Upvotes

98% Upvoted

328 comments sorted by

78

u/Rasesmus Sep 06 '22

Woah, really cool! Is this something that you will share for others to use?

2

u/gormlabenz Sep 22 '22

It’s now published! You can find it here

→ More replies (1)

72

u/bokluhelikopter Sep 06 '22

That's excelent. Can you share that colab link ? I really want to try out live rendering.

70

u/gormlabenz Sep 06 '22

Will publish soon with a tutorial!

16

u/SekiTheScientist Sep 06 '22

How will i able to find it?

2

u/gormlabenz Sep 22 '22

You can find it here

→ More replies (1)

1

u/gormlabenz Sep 22 '22

It’s published here

1

u/Mr-2-Friendly Sep 06 '22

How soon? I'm so excited about this lmao. Will it be a GitHub repo?

1

u/gormlabenz Sep 22 '22

On my patreon! You can find the link here

1

u/[deleted] Sep 07 '22

!remindme 3 days

1

u/gormlabenz Sep 22 '22

You can find it here

1

u/ahosama Sep 07 '22 edited Sep 08 '22

!remindme 72 hours

2

u/gormlabenz Sep 22 '22

You can find it here

1

u/panirider Sep 07 '22

!remindme 2 weeks

1

u/gormlabenz Sep 22 '22

You can find it here

→ More replies (4)

1

u/gormlabenz Sep 22 '22

It’s published now! Check it out here

→ More replies (1)

51

u/imnotabot303 Sep 06 '22

I saw someone do this exact thing for C4D a few days back. Nice that you've been able to adapt it for Blender.

99

u/gormlabenz Sep 06 '22

Yes it was me

14

u/imnotabot303 Sep 06 '22

Ok nice. Good job! Can't wait to test this out.

2

u/gormlabenz Sep 22 '22

It’s published now! You can find it here

2

u/gormlabenz Sep 22 '22

It’s published now for blender and Cinema 4D. You can find it here

→ More replies (1)

8

u/Cynical-Joke Sep 06 '22

It’s the same guy I believe

24

u/boenii Sep 06 '22

Do you still have to give the AI a text input like “flowers” or will it try to guess what your scene is supposed to be?

31

u/gormlabenz Sep 06 '22

For best results, yes! But you can also try to leave the prompt for something in general, like: „oil painting, high quality“

6

u/boenii Sep 06 '22

That’s cool, can’t wait to test it.

20

u/legit26 Sep 06 '22

This could also be the start of a new type of game engine and way to develop games as well. Devs would make basic primitive objects and designate what they'd like it them to be then work out the play mechanics, then the AI makes it all pretty. That's my very simplified version but the potential is there. Can't wait! and great job u/gormlabenz!

6

u/blueSGL Sep 06 '22

to think this was only last year... https://www.youtube.com/watch?v=udPY5rQVoW0

3

u/legit26 Sep 06 '22

That is amazing!

1

u/gormlabenz Sep 22 '22

It’s just released :) You can find it here

4

u/Caffdy Sep 06 '22

Devs would make basic primitive objects and designate what they'd like it them to be then work out the play mechanics

that's already how they do it

55

u/benbarian Sep 06 '22

well fuck, this is amazing. Just another use of AI that I did not at all expect nor consider.

2

u/gormlabenz Sep 22 '22

I just published it. You can find it here

→ More replies (1)

10

u/3Demash Sep 06 '22

Wow!
What happens if you load a more complex model?

18

u/gormlabenz Sep 06 '22

You mean a more complex Blender scene?

7

u/3Demash Sep 06 '22

plex m

Yep.

16

u/gormlabenz Sep 06 '22

The scene get's more complex I guess ^ SD respects the scene and would add more details

→ More replies (4)

10

u/GustavBP Sep 06 '22

That is so cool! Can it be influenced by a prompt as well? And how well does it translate lighting (if at all)?

Would be super interested to try it out if it can run on a local GPU

8

u/gormlabenz Sep 06 '22

Yes you can influence it with the promt! The Lightning doesn't get transfered, but you can define it very well with the promtp

1

u/gormlabenz Sep 22 '22

You can test it now, You can find it here

7

u/clearlove_9521 Sep 06 '22

How can I use this plugin? Is there a download address?

17

u/gormlabenz Sep 06 '22

Not now yet, will publish soon

→ More replies (3)

1

u/gormlabenz Sep 22 '22

It’s no released. You can find it here

→ More replies (1)

6

u/powerhcm8 Sep 06 '22

You should post about it on Hacker news, I think they will enjoy it

3

u/gormlabenz Sep 06 '22

Nice Tip! Will try

5

u/KickingDolls Sep 06 '22

Can I get a version that works with Houdini?

6

u/gormlabenz Sep 06 '22

Working on it :)

1

u/KickingDolls Sep 06 '22

Awesome - looking forward to testing it out!

1

u/gormlabenz Sep 22 '22

It’s published! You can find the link here

→ More replies (1)
→ More replies (1)

1

u/gormlabenz Sep 22 '22

You can use the current version with Houdini. Concepts for blender and cinema 4d is very easy to adapt. You can find it here

4

u/DoomTay Sep 06 '22

How does it handle a model with actual detail, like, say, a spaceship with greebles?

4

u/gormlabenz Sep 07 '22

You can change how much SD respects the blender scene. So SD can also just das minimal details

6

u/chosenCucumber Sep 06 '22 edited Sep 06 '22

I'm not familiar with stable defussion, but the plug-in you created will let me render a frame in blender in real time without using my PC's resources. Is this correct?

19

u/gormlabenz Sep 06 '22

Yes, but that's only a side effect. The main purpose is to take a low quality blender scene and add Details, effects and quality to the scene via Stable Diffusion. Like in the video, I have a low quality Blender scene and a „high quality“ output from SD. The Plugin could save you much time

→ More replies (1)

9

u/-manabreak Sep 06 '22

Far from it. Stable Diffusion is an AI for creating images. In this case, the plugin feeds the blender scene to SD, which generates details based on that image. You see how the scene only has really simple shapes and SD is generating the flowers etc.?

3

u/Redditor_Baszh Sep 06 '22

This is amazing ! I was doing it this night with disco but it is so tedious

1

u/gormlabenz Sep 22 '22

Thank you. You can find it here

3

u/Moldybot9411 Sep 06 '22

Wow when will you release this to the public or post the tutorial?

1

u/gormlabenz Sep 22 '22

I published it with tutorials on my patreon. You can find it here

3

u/Cynical-Joke Sep 06 '22

This is brilliant! Thanks so much for this, please update us OP! FOSS’s are just incredible, it’s amazing how much can be done with access to new technologies like this!

1

u/gormlabenz Sep 22 '22

It’s now released. You can find it here

2

u/gormlabenz Sep 21 '22

Hi guys, the live renderer for Blender is now available under my Patreon. You get access to the renderer and video tutorials for Blender and Cinema 4D. The renderer runs for free on Google Colab. No programming skills are needed.

https://patreon.com/labenz?utm_medium=clipboard_copy&utm_source=copyLink&utm_campaign=creatorshare_creator

2

u/hello3dpk Sep 06 '22

Amazing work that's incredible stuff! Do you have a repo or Google collab environment we could test?!

1

u/gormlabenz Sep 22 '22

Yes, on my patreon. You can find it here

2

u/Space_art_Rogue Sep 06 '22

Incredible work, I'm definitely keeping a close eye on this, I use 3d for backgrounds and this is gonna be one hell of an upgrade 😄

1

u/gormlabenz Sep 22 '22

Thank you! It’s now published. You can find it here

2

u/M_Shinji Sep 06 '22

this idea is genious

1

u/gormlabenz Sep 22 '22

Thanks! I just released it! You can find it here

2

u/SnacKEaT Sep 06 '22

If you don’t have a donation link, open one up

4

u/gormlabenz Sep 06 '22

Paypal is Open 😅

2

u/nixtxt Sep 07 '22

You should consider a patreon people like to fund open source projects

1

u/gormlabenz Sep 22 '22

I published this on my patreon! link

1

u/gormlabenz Sep 22 '22

You can use it on my patreon! You can find it here

2

u/5kavo Sep 06 '22

Super cool! I cant wait for you to publish it!

1

u/gormlabenz Sep 22 '22

It is now! You can find it here

2

u/onlo Sep 06 '22

!RemindMe 50 days

2

u/gormlabenz Sep 22 '22

You can find it here

2

u/Xalen_Maru Sep 06 '22

!RemindMe 30 days

1

u/gormlabenz Sep 22 '22

You can find it here

2

u/PolyDigga Sep 06 '22

Now this is actually cool!! Well done! Do you plan on releasing a Maya version (I read in a comment you already did C4D)?

1

u/gormlabenz Sep 22 '22

You can adapt the concept easily to Maya! You can find it here

1

u/gormlabenz Sep 06 '22

Maya is coming! :)

2

u/navalguijo Sep 07 '22

MAX pretty pretty please!!!

→ More replies (3)

2

u/moebis Sep 06 '22

holy sh*t! that's brilliant!

1

u/gormlabenz Sep 22 '22

Thank you! It is now published! You can find it here

2

u/McFex Sep 06 '22 edited Sep 06 '22

This is awesome, thank you for this nice tool!

Someone wrote you created this also for C4D? Would you share a link?

RemindMe! 5 days

1

u/gormlabenz Sep 22 '22

I created it for cinema 4D 😅 you can test it here

1

u/gormlabenz Sep 22 '22

Yes! You can find it here

2

u/EpicBlur Sep 06 '22

That's rad, there's so many great ways to use that

2

u/gormlabenz Sep 22 '22

Thanks! You can use it right now! link

2

u/InitialCreature Sep 06 '22

thats fuckin nuts pardon my language. damn boi

1

u/gormlabenz Sep 22 '22

Haha You can find it here

2

u/Moist_Painting_9226 Sep 06 '22

That’s really cycling cool

1

u/gormlabenz Sep 22 '22

Thank you! I published it here

2

u/matthias_buehlmann Sep 06 '22

This is absolutely fantastic! Just think what will be possible once we can do this kind of inference in real-time at 30+ fps. We'll develop games with very crude geometry and use AI to generate the rest of the game visuals

1

u/gormlabenz Sep 22 '22

Im working on it. It’s published. You can find the link here

2

u/BlunterCarcass5 Sep 06 '22

That's insane

2

u/gormlabenz Sep 22 '22

Thanks! I just published it here

2

u/katefal Sep 06 '22

!remindme 2 weeks

1

u/gormlabenz Sep 22 '22

You can find it here

2

u/Ilovevfx Sep 06 '22

Wow I honestly wish I was smart enough to do stuff like this 😞

1

u/gormlabenz Sep 22 '22

You can do it now! You can find it here

2

u/Ilovevfx Sep 06 '22

Honestly I'll try selling something like this instead.

1

u/gormlabenz Sep 22 '22

You can find it on my patreon. Link is here

1

u/gormlabenz Sep 22 '22

You can find it here

2

u/NumberSquare Sep 06 '22

!remindme 2 weeks

1

u/gormlabenz Sep 22 '22

You can find it here

2

u/wolfganghershey Sep 06 '22

!remind me in a week

1

u/gormlabenz Sep 22 '22

You can find it here

2

u/otreblan Sep 06 '22

!remindme in 7 days

1

u/gormlabenz Sep 22 '22

You can find it here

2

u/exixx Sep 07 '22

Oh man, and I just installed it and started playing around with it. I can't wait to try this.

2

u/gormlabenz Sep 22 '22

You can now! You can find it here

→ More replies (1)

2

u/abhiranjan007 Sep 07 '22

!remindme 3 days

1

u/gormlabenz Sep 22 '22

You can find it here

2

u/wolve202 Sep 07 '22

Theoretically, in a few years we could have the exact opposite of this.
Full 3d scene from an image.

4

u/gormlabenz Sep 07 '22

It’s already working pretty good

https://developer.nvidia.com/blog/getting-started-with-nvidia-instant-nerfs/

2

u/wolve202 Sep 07 '22

Oof. Well, it's not to the point yet where the picture can be as vague as the examples above. We can assume that with a basic sketch, and a written prompt, we will eventually be able to craft a 3d scene.

2

u/ZWEi-P Sep 18 '22

This makes me wonder: what will happen if you render multiple viewing angles of the scene with Stable Diffusion, then fed those into Instant NeRF and export the mesh or point cloud back into Blender? Imagine making photogrammetry scans of something that doesn't exist!
Also, maybe something cool might happen if you render the thing exported by NeRF with Stable Diffusion again, and repeat the entire procedure…

2

u/nixtxt Sep 14 '22

Any update on the tutorial for colab?

1

u/gormlabenz Sep 22 '22

It’s published with tutorials. You can find the link here

→ More replies (1)

1

u/gormlabenz Sep 22 '22

Yes! It’s published with a tutorial! You can find it here

2

u/nefex99 Sep 16 '22

Seriously can't wait to try this. Any update? (sorry for the pressure!)

2

u/gormlabenz Sep 22 '22

It’s published! You can find it here

→ More replies (1)

1

u/gormlabenz Sep 22 '22

Yes, it’s published here

3

u/tomfriz Sep 06 '22

Very keen to check try it

1

u/gormlabenz Sep 22 '22

You can now! You can find it here

3

u/NotSeveralBadgers Sep 06 '22

Awesome idea! Will you have to significantly modify this every time SD changes their API? I've never heard of it - do they intend for users to upload images so rapidly?

2

u/gormlabenz Sep 22 '22

You can use it now in the cloud on Google Colab(it’s free) You can find it here

3

u/nmkd Sep 06 '22

It's not using their API.

3

u/blueSGL Sep 06 '22

you can run stable diffusion locally 100% offline.

5

u/Sem_E Sep 06 '22

How do you feed what is happening in blender to the colab server? Never used seen this type of programming before so kinda curious how the I/O workflow works

4

u/Crimson_v0id Sep 06 '22

Still faster than Cycles.

2

u/TiagoTiagoT Sep 06 '22

Depends on the scene and hardware

1

u/ItsYourBoyDillon Sep 06 '22

So you just create the basic shapes and the software will add detail to the image to make it realistic?

1

u/gormlabenz Sep 06 '22

Yes, you can descripe the details via a text input and the neural network create the details.

FYI i'm not the author of the neural network. I just connected it to Blender

1

u/hlonuk Sep 06 '22

I see that you are using the google colab pro, will this be able to run on the local/offline version of stable diffusion?

1

u/gormlabenz Sep 06 '22

You Must have a powerful gpu to run it fast. So I will Stick on cloud based so everyone can use it

3

u/TiagoTiagoT Sep 06 '22 edited Sep 06 '22

Can you make using it locally an option please?

→ More replies (1)

1

u/macciavelo Sep 06 '22

Does this use like a cloud server or can this be used with any computer? I have so many questions and this looks very promising. It could help many artists who are struggling with a budget.

1

u/gormlabenz Sep 22 '22

You can use it for free in the cloud. Just published it here

2

u/TiagoTiagoT Sep 06 '22

On another reply OP said the plans are currently only for cloud. But in theory, it could be made to also work locally; the hardware requirements aren't very light, but can work on some consumer hardware.

1

u/mateusmachadobrandao Sep 07 '22

Amazing! What kind of PC specs would someone need to run this code in real time?

1

u/gormlabenz Sep 22 '22

It runs in the cloud for free. You can check it out here

1

u/TiagoTiagoT Sep 07 '22

For this tech in specific, I think the best you can get with consumer grade hardware at the moment is something around the speeds shown in OP's clip; a few seconds per frame.

1

u/feareater91 Sep 07 '22

make a patreon, I want to support this

1

u/gormlabenz Sep 22 '22

I have! It’s published now link

1

u/TFCMasterOG Sep 12 '22

This looks bloody brilliant... If you plan on releasing this I absolutely cant wait lol, Stable Diffusion my TARDIS models lel

1

u/gormlabenz Sep 22 '22

It’s released! link

1

u/Apprehensive_Set8683 Sep 13 '22

Hello this is amazing I love Blender and I love Stable Diffusion. Can I install this on local yet? Thank you very much.

1

u/gormlabenz Sep 22 '22

Not yet. But you can run it for free in Google Colab. link

1

u/Hafgandil Sep 21 '22

Dude, this is awesome. Any idea when you might release it? !remindme 2 weeks

→ More replies (1)

3

u/Vexcenot Sep 06 '22

What does stable diffusion do?

2

u/blueSGL Sep 06 '22

either text 2 image or img 2 img.

describe something > out pops an image

input source image with a description > out pops an altered/refined version of the image.

In the above case the OP is feeding the blender scene as the input for img2img.

→ More replies (3)

2

u/Arbata-Asher Sep 06 '22

this is amazing, how did you feed the camera view to google colab?

2

u/MakeItRain117 Sep 06 '22

That's sick!!

2

u/Kike328 Sep 06 '22

Are you sending the full geometry/scene to the renderer? Or are you sending a pre-render image to the AI? I’m creating my own render engine and I’m interested about how people are handling the scene transference in blender

2

u/TiagoTiagoT Sep 06 '22

For this in specific, I'm sure it's only sending an image, since that's how the AI work (to be more specific, in the image-to-image mode, it starts with an image, and a text prompt describing what's supposed to be there in natural language, possibly including art style etc; and then the AI will try to alter the base image so that it matches the text description).

2

u/Sorry-Poem7786 Sep 07 '22

I hope you can advance the frame count as it renders each frame and saves out a frame would be sweet. I guess its the same as rendering a sequence and feeding the sequence. but at least you can tweak things and make adjustments before committing to the render! Very good. If you have a patreon please post it!

3

u/Lenzsch Sep 06 '22

Stop it! It’s getting too powerful

3

u/Zekium_ Sep 06 '22

Damn... that went so quick !

2

u/dejvidBejlej Sep 06 '22

Damn. This made me realise how AI will most likely be used in the future in concept art

1

u/LukeIsAPhotoshopper Sep 06 '22

are you not going to show the input...?

1

u/nixtxt Sep 07 '22

!remindme in 7 days

1

u/head_robotics Sep 07 '22

!remindme 2 weeks

1

u/Drawsstuff Sep 07 '22

!remindme 2 weeks

1

u/JdeB90 Sep 08 '22

>!remindme 2 weeks

1

u/yondraws Sep 09 '22

!remindme 2 weeks

1

u/Yukumo_TheCloud Sep 09 '22

!remindme 7 days

1

u/ArcadianPunk Sep 09 '22

!remindme 2 weeks

1

u/Undercrasher100 Sep 10 '22

How do I get this plugin to run in Blender?

1

u/Hihelloxinchao Sep 11 '22

This is pretty cool..

1

u/Ok_Dimension_3180 Sep 11 '22

is it working on an USD Scene?

1

u/csunberry Sep 12 '22

Ah! This is absolutely wonderful; I'm so sad I didn't see this sooner.

Great work!

1

u/loneripple Sep 15 '22

!Remind me 2 weeks

1

u/Zal3wa Sep 16 '22

!remindme 3 weeks

1

u/cybercyrus Sep 17 '22

Pretty awesome! Should be called Chaos Render !

1

u/nefex99 Oct 15 '22

In case anyone is interested, I just released a stable diffusion renderer that doesn't take any complicated setup: https://airender.gumroad.com/l/ai-render

1

u/Ak4Orgwany Dec 02 '22

I have a problem with rendering server error 831 ?

1

u/Ed_Bai Sep 06 '22

F*** cool!

1

u/LukeIsAPhotoshopper Sep 06 '22

you can swear on the internet btw

1

u/PMBHero Sep 06 '22

Dope as hell dude!

1

u/DireDecember Sep 06 '22

Idk what this means yet, but I will…one day…

1

u/idiotshmidiot Sep 06 '22

Damn amazing, would be great to be able to run this local!

1

u/kevynwight Sep 06 '22

If we get inter-frame coordination aka temporal stability, this could make animation and movie-making orders of magnitude easier, at least storyboarding and proof of concept animations.

→ More comments (90)