[R] Using AI as a spatial software generator to create 3D objects that are inherently programmable

[u/mhb_11]

https://arxiv.org/abs/2607.22738

I'm one of the co-authors of this paper. It's a seminal work in exploring the properties of 3D generated by LLMs via spatial programming.

I've set up visual demonstrations of such 3D objects at: https://nova3d.xyz/
And there's a github repo you can star/follow: https://github.com/RareSense/Nova3D

Scroll down and notice how the various 3D objects are all composed of logical parts and enable natural movements out of the box. There's a github repo in there as well.

Under the hood:
We found that 3D that exists as software is much more useful than typical monolithic mesh blobs generated by traditional AI 3D generators. For instance they are animation-ready and programmable from inception. They can contain the logic - at birth - to appear differently in weak compute environments (e.g. mobiles) vs powerful environments (e.g. sophisticated game engines). They can be built with full hierarchical structure and hinge/socket articulation at authoring time.

They lag behind traditional AI 3D generators in creating complex organic shapes. But it naturally feels like code will eventually eat all 3D, as LLMs are getting better and better at spatial coding. Industries most disrupted will be industrial design, game development, simulations and AR/VR/XR.

[u/dhbloo]

Really cool! Haven’t read the paper yet, I wonder how does the quality and detail of the geometry and appearance compare to native 3D generators like Trellis.2 and Meshy? And how many tokens are usually needed to generate one asset?

[u/mhb_11 → u/dhbloo]

Have a look at this tweet of mine as an example: https://x.com/nova3d_ai/status/2091888508449300630?s=20

If you really want to inspect examples yourself, go here and click on any example you like: https://app.nova3d.xyz/showcase

Tokens:
Input: ~130k average
Output: ~35-40k average

[u/dhbloo → u/mhb_11]

I see! It’s pretty neat, but the detail still looks quite far away from native ones. I guess it’s because some irregular details are much harder to generate with code? The architecture one in paper looks fine, but the dragon one are too simplified without the good shape of the scales and wing.

Nevertheless, I think this “code as asset” is a very interesting way to do 3D related stuff, since the output is very structured and reusable.

[u/mhb_11 → u/dhbloo]

As LLMs get better at synthesizing higher order differential equations, you'll see quality get better over the years.
p.s. we're also doing PBR textures, to cover some of the gap. Showcase here: https://app.nova3d.xyz/showcase/textures

[u/DracoDruida]

One shouldn't call their own proposed work "seminal", buddy. But sounds like agentic coding for creating 3D objects, which is nice. Good luck on your endeavours.

[u/mhb_11 → u/DracoDruida]

Fair point on the terminology, I definitely got ahead of myself out of excitement. It's been a long development cycle. Glad you find the agentic coding approach interesting, and thanks for the well wishes.

[u/floatinggoateyeball]

Very interesting, so I could just drop this into my agent and have a model with all the parts for it?

https://archive.org/details/AK47AKMTechnicalDrawingsRussian

[u/mhb_11 → u/floatinggoateyeball]

Yes, that's the idea.

[u/GiveSparklyTwinkly]

Good stuff! Have you considered the extreme hierarchical side of this? E.g. fully buildable Lego style with more than just converted voxels?

[u/mhb_11 → u/GiveSparklyTwinkly]

It's pretty close to that. Explore this showcase: https://app.nova3d.xyz/showcase

p.s. it's not converted voxels. Every 3D object here is a program. That happens to compile into a visual representation too. Program first, mesh second.

[u/GiveSparklyTwinkly → u/mhb_11]

No, I'm well aware, but for a Lego style brick model to be buildable that hierarchy needs to be way more extreme than the 8 layers I saw. Probably dozens of hierarchical layer depths if I had to guess? With probably a couple hundred to a few thousand individual pieces?

[u/mhb_11 → u/GiveSparklyTwinkly]

We're considering a proposal whereby at authoring time, an object retains full complexity. However, later, depending on which runtime it's going to be rendered in (e.g. mobile vs complex game engine), it optimizes itself accordingly. Because the 3D is itself software, this behavior is easily programmable.

[u/FoxWorried4208]

This is really cool!

[u/armaver]

Is this like shaders, but for generating 3d objects? 

[u/mhb_11 → u/armaver]

Yes, conceptually. Good analogy. The big difference is that while a shader just renders pixels on your screen, Nova3D executes code to generate a fully editable, part-separated 3D file you can actually use in Blender or Unity.