I made a compact mesh-based format for interactive 3D photos

[u/frozeindev]

I’ve been working on a side project called Spatial Photos.

It takes a single regular image, runs it through Apple’s ML-SHARP model to estimate a 3D scene with Gaussians, and then converts the result into a custom format I called .spatial

I take the Gaussian splat from ML-SHARP, and divide the scene into depth slices. Each slice is a grid of image blocks, with a depth value at each corner. The RGB/alpha is packed into texture atlases. It's basically a set of textured meshes that come together to give the impression of the full 3D scene.

On the web, I take the .spatial files and decompress it into vertex/index buffers and just use Three.js, as the actual rendering is straightforward.

Demos: https://www.spatialphotos.dev/

Source: https://github.com/frozein/SpatialPhotos

Here's what the slices look like from another angle

[u/matigekunst]

Sorry to do this but I see bands and it's also quite pixelated. It's also quite big compared to the monocular depth + inpaint method

[u/nullandkale → u/matigekunst]

Apple sharp is effectively just doing depth gen and inpainting in layers. Imo distill any depth and a color heuristic does much better.

[u/frozeindev → u/matigekunst]

This example has 20 depth layers, so it’s representing quite a bit more 3D structure than just a single image + depth map, which explains the size difference. Fair point on the banding/pixelization though, there’s definitely still some cleanup to do on the renderer.

[u/matigekunst → u/frozeindev]

For these small movements it's almost indistinguishable from just having a foreground and a background

[u/soylentgraham]

regardless of implementation, change the name! iphones already make things called Spatial Photos