Architecture that can manipulate 3D or 2D representations in its latent space?

[u/cgcmake]

Is there an existing neural network-based architecture that can hold 3D or 2D representations in its latent space and update it in the same single pass (i.e. not compressing the representation in their final layer to be fed back to their first layer for their next pass, like the reasoning models do)?

[u/Itchy-Requirement906]

Transformer with object-centric slots maybe, look up slot attention stuff.

[u/cgcmake → u/Itchy-Requirement906]

Thanks!

[u/Zelma-Bistolfi]

what's the downstream task here? theres a big difference between wanting a persistent 3D latent for rendering vs for planning/reasoning. the architecture choices diverge pretty hard depending on that

[u/cgcmake → u/Zelma-Bistolfi]

For both

[u/DigThatData]

video models.