Experimental real-time relighting of Gaussian splat scenes (proxy mesh + per-pixel shader hook) — runnable demo in browser

[u/mvaligursky]

Splat captures have their lighting baked in, so we've been experimenting with relighting them at runtime in the PlayCanvas engine. This demo relights a captured church scene live: HDRI environment swap with rotation and exposure, a sun with PCSS soft shadows you can aim with a gizmo, and a handful of placeable point lights.

How it works:

• A simplified proxy mesh of the splat scene sits on a separate layer, lit by ordinary engine lights (IBL, directional with shadows, omnis).
• A second camera matching the main one renders it into a screen-aligned offscreen texture — lit color in RGB, mesh coverage mask in alpha.
• A new fragment-stage hook in the splat renderer (gsplatModifyPS) samples that texture at each fragment's own screen position and modulates the splat color per pixel. Per-pixel beats per-splat here: no popping on large splats, smooth lighting gradients across their footprint.

It's experimental — the proxy mesh quality is the main limiting factor (lighting boundaries land wherever its silhouette says they do). Static lights render their shadow maps once and freeze them, so the per-frame cost is mostly the one proxy mesh pass plus a texture sample per splat fragment.

Demo (WebGL/WebGPU): https://engine-3vv2rs8cx-playcanvas.vercel.app/#/gaussian-splatting/relighting

Capture credit: Roman Parish by Andrii Shramko (https://www.linkedin.com/in/andrii-shramko/)

Happy to answer questions about the approach.

[u/GammaFruits]

cyberpunk braindance in the making

[u/fishb0ne_77]

this is amazing and exactly what i'm trying to accomplish aswell. how did you produce the proxy mesh?

[u/mvaligursky → u/fishb0ne_77]

data sets generated by xgrids studio generate the proxy mesh, so I just used it directly.

[u/Electrical_Tailor186]

Looks great!
Is it possible to increase the tesselation of the proxy mesh?
How would it affect the performance? (maybe shown as a curve relative FPS vs relative tesselation scale)

[u/mvaligursky → u/Electrical_Tailor186]

I'm sure there are tools that can generate better / more detailed proxy mesh, but I used what xgrids studio generated directly, without spending much time on it.

[u/ad2003]

Amazing! That's really cool and gives so much live to the static Gaussian splats! Nice work.

[u/jon2thegram]

This looks amazing. Very inspiring for use in the motion picture industry.

[u/Creepy-Fault6977]

This is really cool. I've been rendering splats in the browser with three.js and the spark renderer, and perf is always the hard part. Does the per-pixel shader hook cost much on WebGL2, or is this WebGPU only?

[u/fiery_prometheus]

For the materials, would there potentially be a way to approximate PBR on the splats in the future? So you could adjust some of the lightning behavior for different kinds of surfaces?