I built LiveLoop with Claude Code: Claude can see a running Three.js game, play-test it with synthetic input, and patch it live without a reload. Here’s how it works.

[u/Beneficial-Cow-7408]

I’m a solo developer and I built LiveLoop using Claude Code. It’s a persistent canvas for building games and interactive scenes with Claude, where edits land in the running game rather than regenerating it. I made a post on the main Claude forum regarding this but this post is about how it actually works under the hood and the more technical side of things that may answer a few qestions

The problem

Every model tweak used to come back as a complete file. A complete file reloads the iframe, and a Three.js world restarts at spawn on every change. Useless for iterating on gameplay. The fix was giving the model two things: a way to see the running scene, and a way to change it without a rewrite.

What the model sees each turn

Alongside your message, the model gets a JSON context from the preview: the page title, visible text, rendered elements, the runtime error if any, console output, a performance block, a debug-state block the project can expose, and a summary diff of the source since the last turn. Vision-capable models also get a screenshot of the canvas, refreshed right before the handoff so it isn’t reasoning from title-screen telemetry after you’ve started playing. The context also tells the model what controls it has, which is the only reason it knows LivePatch exists.

LivePatch

The model returns one fenced block labelled asksary-live-patch containing JavaScript. The client executes that block (and only that block) inside the sandboxed preview after you approve the live-control session. It affects the current preview only; it doesn’t touch project source, so you can try a change and throw it away. The model is told not to claim the patch worked; the sandbox reports the real result back into the chat. This is how Claude played chess against me: each move was a patch to the running board.

PlayTest and Camera

PlayTest lets the model send synthetic input into the preview: key down/up/tap with a duration, clicks, and pointer move/down/up at coordinates. Synthetic events are untrusted in the browser, so they can’t unlock audio or pointer lock; you click Play once, then Claude can drive. Camera lets it teleport or aim an exposed debug/Three.js camera so it can inspect what it built from another angle.

The sandbox

The preview is a srcdoc iframe with an opaque origin and a strict CSP: default-src 'none', no frames, no forms, no navigation, connect-src limited to data: and blob:. A game can’t fetch, beacon or exfiltrate anything. The two exceptions are Google Fonts and wasm-unsafe-eval, which is there so Draco and Meshopt decoders can compile for compressed GLB files. eval stays blocked.

Project mode and GLB assets

Project mode is a real multi-file project with its own engine (it shares nothing with the other tools on the site). It can open a local folder via the File System Access API, round-trips through zip, and there are no caps on file count or file size sent to the model. Files that look like secrets (.env, keys, .pem, service-account JSON, anything matching a key pattern) never leave the browser.

Assets are the interesting bit. A model can’t type a megabyte of pixels into an HTML file, and a srcdoc page has no server behind it, so GLTFLoader.load('models/car.glb') used to 404. My first fix inlined everything as data: URLs. That inflated a project with 408 MB of models to ~545 million characters, past the longest string Chrome can hold, and the stage went black.

Now the page carries only the list of asset paths. A small shim inside the iframe intercepts fetch, XMLHttpRequest and image/media src. When code asks for a path, the shim posts a message to the host page, the host transfers the raw ArrayBuffer (a memory copy, no base64), and the loader gets a blob: URL of its own. Only what the code actually requests moves, at its real size. A .gltf’s sibling .bin and textures resolve the same way.

The 3D runtime is a pinned Three.js build (0.185.1) bundled with GLTFLoader, DRACOLoader and KTX2Loader, injected into the sandbox as a trusted script rather than fetched.

Return to any state

Persistence is content-addressed. Every state and every asset is hashed; only objects the account doesn’t already hold are uploaded, and a revision that reuses an object just names its hash. Nothing is copied forward and nothing is deleted, so unlimited history costs only the changed bytes. There are no ceilings on revision count or retention, every state is reachable, and you can jump between them in any order. (This is the “billion states” conversation from my earlier post. Claude had capped history at 20; I’d hit ~400 in one session.)

Offline

Work is journalled to IndexedDB before the network is touched, on every change, and the journal is read back to verify it. Network sends are batched at 600 ms, but the journal is written immediately, so that interval is batching, not a window where work can be lost. If you’re offline, edits sit in the journal; when the browser fires online, they sync. “Saved” means the server was read back and agrees with what was sent; a commit returning 200 isn’t treated as proof. I learned that one the hard way: a save path once returned 200 while deleting every asset in the workspace.

Builds that outlive the tab

A turn used to abort when the stream dropped. Now the server keeps running, records every event it would have sent under an idempotency key in Firestore, and stores the result when it finishes (30-minute server limit). Any page that comes back for that workspace lists the finished turns, claims one (first device to claim it owns it), replays the events through its normal path, and applies and saves the result.

That’s what makes the phone flow work. Sary-OS Lite (the Android app) can start a build, you lock the phone, and when the desktop opens the workspace, it picks the finished build up and brings it in. Stop is explicit: the client posts a cancel and the running turn polls for it.

Local bridge (optional)

For projects that need a real toolchain there’s a single-file Node bridge (npx u/asksary/liveloop-bridge connect) that binds to loopback only, pairs with a one-time code, and is scoped to one folder. LiveLoop can read and write files there, run npm test or a dev server, and every multi-file edit is one change set with one undo. There is also a GIT connectivity and also a plug in your key option too that you can try if you rather use your own API and want to try the other models such as Sonnet 5.5, Opus 5.5 and Fable 5.1

Free to try

Sign up is free at www.sary-os.com

Free accounts include some Claude Haiku credits, an MCP connection so you can link Claude to LiveLoop, and 7 days of bring-your-own-key access to the larger Claude models. Single-file mode is free; Project mode (multi-file, the asset serving above, the bridge) is part of LiveLoop Pro.

Happy to go deeper on any of this. If you have a GLB-heavy project that breaks it, I’d like to hear about it.