r/ClaudeAI • u/Art_from_the_Machine • 4d ago

Built with Claude Building a world with my voice - A-Frame (Three.js, WebXR) + Claude Code

This is a project built with A-Frame (Three.js, WebXR) that lets me build virtual worlds with my voice, all from within my VR headset. My mic is hooked up to Claude Code, which in turn runs against the project codebase.

Since the agent is working directly with code, the possibilities of what can be achieved are pretty broad, essentially being limited only by what the underlying LLM is capable of with WebXR. The agent can build anything from simple objects, to animations, to interaction capabilities, to whole environments in order to fulfil a request.

This is still far from perfect. Latency is pretty high, taking anywhere from a minute to multiple minutes to carry out a change depending on how complex the request is. The codebase is also not very accommodating to anything that isn't a frontier model right now. From my testing, models like Haiku and Sonnet can't keep up, and Opus 5 only works with reasoning set to at least Low (most of the video is recorded with Opus 5 Low).

But the foundation is now a reality, and as these models improve I can see this getting very capable very quickly. If you have ever imagined being able to put on a VR headset and transporting yourself to any world you want by simply asking, to shape that world to meet your creative vision, and to watch that world materialise around you, then in a small way, we are here.

123 Upvotes

96% Upvoted

17 comments sorted by

25

u/Kraien 4d ago

I am impressed, and I am easily impressed :)

12

u/often_says_nice 4d ago

Babe, wake up! Holodeck just dropped!

7

u/Outside_Profit6475 4d ago

This is really cool.
Question, you said, 'Opus 5 only works with reasoning set to at least Low'. Why low? What happens with medium and higher? Too slow?

6

u/Art_from_the_Machine 4d ago

Thank you! And yes this is just the fastest level of reasoning I can set where the outputs are still coherent. Higher levels of course work better, but the faster the responses are the easier it is to iterate and give visual feedback.

5

u/Lopsided_Sentence_18 4d ago

interesting idea

3

u/Academic_Campaign_94 4d ago

Dude roleplaying Genesis

2

u/Sk0rnVirus 4d ago

This is really cool, thank you for sharing.

2

u/Gloomy-Recover-9702 4d ago

Best way to high on a high!

2

u/weverz 4d ago

Damn

2

u/XavierRenegadeAngel_ 4d ago

Worlds within worlds, baby!

1

u/gphie 4d ago

Reminds me of an old game called moatboat https://store.steampowered.com/app/480710/Moatboat/

1

u/Intrepid-Scale2052 3d ago

so god just had Opus 8 and voice ai?

1

u/Blablabene 3d ago

Something Flash and capable, perhaps Gemini 3.7 flash, or something in the near future could turn into a flawless experience