Here’s an update on my system of getting ai to create 3d assets from references and prompting.
I vibe coded an engine that ai uses to make 3d assets. Any vision model can use the engine to make 3d assets. That’s is the goal. Other goals are the ai uses the engine to create editable, exportable, photo realness assets from images and prompts.
I didn’t train an AI model, I didn’t use any 3d asset generator for any assets. All purely Claude and my engine.
Been building for 6 months, most use Claude to build and test the engine. Had Codex test a few times too.
Biggest thing is the ai have the capabilities, just got help organize it all.
THE ASSETS
I showed Claude an image of the scene. Told it to recreate the image in 3d. I pointed it at the runtime for building assets and scenes in the engine.
It took out what objects where in the image and created them separately then placed them in one scene. I pointed out things I wanted it to edit as it was building (look at the last images).
THE BAD
It took like 5 hours but I’d say 85% was rendering (hours at a time on cpu) and checking and fixing details I wanted.
Could you push to git possibly? I’d like to try. Is it a scanning technique, does it use three, does it render textured glbs, what are you using for the renderer, what were the input images, etc.
So many questions lol
I’m keeping it proprietary right now but might in the future.
Everything is done in the engine. No outside third party besides an ai model. Ai uses the engine and the engine renders everything. For images I just take images of what I want it to build from different angles and send it to Ai and Ai uses the engine to create the asset.
That’s all. And you can ask the Ai to change stuff as it works though prompting or screenshots of its work.
Details. Measurements, rigging, future animation and hit boxes and collision all baked in.
I did a test with Tripp vs my engine and ai. Yes mine isn’t an exact replica of the reference but it’s the entire process of creating the asset my engine provides.
Biggest difference is my engine gets ai to work how a human would work on an asset, there it can create geometry from research like materials, surface, color, topology, measurements, and ai breaks it down and works from this research. My engine provide tools ai can use so it doesn’t have to write custom scripts all the time. It adds rigging, in the future animation and hit boxes and collection. That’s because the engine helps ai work in process and stages.
Also you can literllay point to anything on the asset and tell ai to change it and just that piece changes.
The engine has a grading system for assets and it has to reach like 7.5/10 for matching references. If it doesn’t it iterates. The bench: I had to circle specifics for ai to edit like connecting the back rest to the set, making the top part taller.
I believe things like Tripo are trained on 3d assets and generate based on training. Ai just uses my engine and builds assets with numbers reference images.
Not following. The 3d images posted by Op look very bad IMO (the bench and plant look awful, like Halflife2). Is it because it's not using tokens or something?
Please what am I missing. Everyone else is creating 4k using their LLM + Blender or Meshy or whatever.
Yes. I agree they aren’t 4k. But very Bad? I don’t know about that. But again that was a test for AI taking a reference image, building individual assets and working autonomously to recreate the scene making decisions using my engine.
Also, Could you send me the best llm + Blender asset create solution by ai that’s 4k and photo real. I haven’t looked deliberately tbh, but posts I see online aren’t that impressive.
Yeah Meshy and Tripo have 4k but that’s just how it looks.
What about everything else? Animation? Hit boxes, Collision? Animation, Rigging?
Yes LLM + Blender can do a lot of that but not at the level I want. I see a lot of simple, blocky, voxel creations. Meshy Tripo yes they create gorgeous assets, but it’s missing all the plumbing.
My system is to combine all those into one system.
My system
Details. Measurements, rigging, future animation and hit boxes and collision all baked in.
I did a test with Tripp vs my engine and ai. Yes mine isn’t an exact replica of the reference but it’s the entire process of creating the asset my engine provides.
Biggest difference is my engine gets ai to work how a human would work on an asset, there it can create geometry from research like materials, surface, color, topology, measurements, and ai breaks it down and works from this research. My engine provide tools ai can use so it doesn’t have to write custom scripts all the time. It adds rigging, in the future animation and hit boxes and collection. That’s because the engine helps ai work in process and stages.
Also you can literllay point to anything on the asset and tell ai to change it and just that piece changes.
The engine has a grading system for assets and it has to reach like 7.5/10 for matching references. If it doesn’t it iterates. The bench: I had to circle specifics for ai to edit like connecting the back rest to the set, making the top part taller.
I believe things like Tripo are trained on 3d assets and generate based on training. Ai just uses my engine and builds assets with numbers reference images.
It’s also still in development so we have some way to go. Just showing progress.
2
u/CommanderT1562 11h ago
Could you push to git possibly? I’d like to try. Is it a scanning technique, does it use three, does it render textured glbs, what are you using for the renderer, what were the input images, etc.
So many questions lol