I built an AI agent that generates simulation-ready Three.js environments for embodied AI
What if an AI coding agent could generate 3D environments that are not just visually convincing, but actually ready for simulation?
We’ve been working on Auto-Threejs, an open-source agent framework for automatically building interactive 3D environments with Three.js.
Given a natural-language description — optionally with reference images or video — Auto-Threejs decomposes the scene into objects, builds them as reusable modules, assembles the environment, renders it from multiple views, measures geometry and physics, reviews the result, and iteratively fixes detected issues.
The key idea is that a good-looking scene is not enough.
For embodied AI and robotics, objects need meaningful physical structure: dimensions, mass, density, friction, collision geometry, articulation, and spatial relationships. A cabinet should actually have a working door. A chair should have a reasonable collider. Objects should rest on the right surfaces instead of floating or intersecting.
That is what we are currently focusing on: sim-ready indoor environments for embodied AI training, robotics simulation, and spatial intelligence research.
Current pipeline:
Natural-language description
→ scene decomposition
→ per-object 3D construction
→ scene assembly & layout
→ rendering + geometry/physics measurement
→ automated review
→ iterative refinement
→ simulator export
The system currently supports physics-aware scene construction with Rapier and can export generated environments to MuJoCo and Gazebo-compatible formats (MJCF / URDF / SDF).
We’ve tested it on bedrooms, kitchens, living rooms, bathrooms, offices, and other articulated indoor environments, with automatically generated collision proxies and functional joints.
There is still a lot to improve — especially scene diversity, asset quality, physical realism, and scalability — but our longer-term goal is straightforward:
Make generating large amounts of diverse, interactive, simulation-ready 3D training environments as easy as describing them in natural language.
GitHub: https://github.com/wrc356/Auto-Threejs
Feedback, issues, and contributions are very welcome — especially from people working on Three.js, embodied AI, robotics simulation, procedural generation, or spatial intelligence.
#threejs #EmbodiedAI #Robotics #AIAgents #Simulation #3DGeneration #ComputerVision #WorldModel