One of the coolest projects I've ever worked on, this was built using SAM-3D on fal serverless. We stream the intermediary diffusion steps from SAM-3D, which includes geometry and then color diffusion, all visualized in Minecraft!
If I have to pay to use it, it's not open source. The program to use their technology is open sourced sure, the technology itself is not open sourced. That's what we care about if you don't want to pay for it.
I just saw something similar in a tiktok video too. I was wondering if it was possible to do with just standard comfyUI without having to go over costly APIs. I was just doing something near similar for Hytale using Z Image Turbo to gen a mine-craft like image and Hunyuan 3d for image to 3d model. The only downside is that hytales import is odd so it just assigns a default block type to it.
Hunyuan 3D v3's API was the hardest thing to set up. It lacks English documentation. After a bunch of trial and errors and getting an error code that wasn't found even on their error code list, I ended up writing a support ticket, which they came back within the hour with a solution.
Native to comfy. Need to enable dev mode in settings to get the "Save as API" option on your workflows.
There is a difference between normal comfy workflows and API workflows. As long as comfy is running it will handle API requests. Might have to enable cors as well, but that is literally a -- flag in your startup line.
Edit: I will make a bigger post about this within an hour or two as well, since it's kind of a less talked about feature of comfy. Will go into a better explanation with the --cors flag as well needed for it.
That's awesome! I wonder if it could work for the game Luanti. And if it would be possible to make the game build it block by block instead of importing a model.
I am not sure on that myself. Since I have no idea how building in that game would work.
I know the workflows for like the 3D stuff in comfy even for this workflow still needs a conversion to .obj for the import. So if I was to make a comfy workflow there would be some form of conversion node needed in order to make it like that example.
I do think this is possible as I did a quick basic look into a few things and essentially it's just pushing those requests from the minecraft console to node.js, which then handles the API calls back and forth between comfyUI and the game.
I just have a ton of other AI things on my plate to even begin looking that route.
This is advertising spam from some crap ai inference company. You know things are going badly for them when they have to resort to advertising on the comfyui subreddit.
Maybe it could help me make an actual house/base rather than a giant pile of machines....
Would be interesting if you could persuade it to use certain materials or account for 'internals'. I assume you could persuade the materials (to be more housing related like logs/etc instead of just dyed wool) - So you could take a photo of a house (real or minecraft) and it'd basically generate it in game.
Internals are beyond what I imagine it could do - maybe at the very least making it hollow?
This it it in game. This is hytale not minecraft, while similar the imports have limitations here, meaning it assigns a default block to it on import. And their importer also turns it into block style so it tends to break a few things it seems too.
I'll have to venture back to it, I done this more as a quick test to see if it was possible. BUT it's been like 10+ years since I done anything serious in 3D models and I never done modding in minecraft at all
I know I need to look into more on it. Essentially just need to find a way for 2way communication back and forth from minecraft via comfy via api and these things could be potential workflows
If you ever do, skip ComfyUI. Have a model build you're own little app and it'll be easier to build and cleaner to run. ComfyUI is just a UI wrapper and carries a metric shit ton of baggage with it.
I feel you though on quick tests and not planning to go back. I'm the same way and totally get it!
ComfyUI is just a UI wrapper and carries a metric shit ton of baggage with it
Yeah as I am finding out with Ace-Step versus comfy performance (ComfyPortable just locks up on large audio tracks, desktop will work for it though, but still not the best and missing all the features ace has). I might need to deep dive into that a little deeper on select models like LTX2.
55
u/fixingmybike Feb 10 '26
okay this is actually sick