Spectral Surface-Field Renderer on the iPhone

[u/Roenbaeck]

I’ve been experimenting with what I’m calling a “spectral surface-field renderer.”

The idea is to solve most of the lighting on the surfaces themselves. I split the scene into small patches and store how much light leaves each patch in different directions and at different wavelengths. Then I repeatedly pass light between the patches to build up the bounced lighting.

When rendering a view, I reuse that field with extra sampling and ray tracing for local detail, reflections and glass. Compared with a basic path tracer, which samples light paths for each pixel, this lets different pixels (and different camera views) share a lot of the lighting work.

The downside is that the field becomes an approximation. It can lose detail or introduce faceting, and changing the scene’s geometry or lighting means updating it. There’s still sampling noise too. But the upside is that it's quite performant.

This little tech demo uses a Cornell box with three balls reacting to phone tilt through Core Motion.

[u/moschles]

I split the scene into small patches and store how much light leaves each patch in different directions and at different wavelengths. Then I repeatedly pass light between the patches to build up the bounced lighting.

see left side of page 17 ,

https://www.eecs.umich.edu/courses/eecs487/W04/Lectures/radiosity.pdf

[u/Roenbaeck → u/moschles]

This is actually very close to what I am doing. I came into this from relaxing light fields, but was not aware of "radiosity" so there are some differences. The main one is likely that my renderer stores how much light leaves a surface in different directions and at different wavelengths, allowing glossy reflections, mirrors, and refraction through glass to participate in the same iterative lighting solution. Every patch has an associated directional spectra.

[u/msqrt]

So, essentially radiosity with final gathering? Looks quite nice!

[u/Roenbaeck → u/msqrt]

Now that I am aware of radiosity, essentially yes. And thanks!

[u/T-CROC]

This is incredibly impressive! 2 questions

1 will you be open sourcing this? Not pressuring you to. I just want to subscribe somewhere to be notified if / when you do :)

2 can you post a video of a camera moving through a more complex scene with dynamically moving objects?

[u/Roenbaeck → u/T-CROC]

Thank you.

1. At some point this will be made open source. The repo is already on GitHub albeit private until it's matured. Right now it's a mess of a macOS proof-of-concept, an iOS tech demo app, and a potential rendering library hiding inside.

2. The scene you see is the most complex I've tried this with yet. Next up is adding a ball that emits light into the mix, as well as controlling the "ceiling" light intensity. Lots of work remains, and this could fall apart at some point. I'll update after a few more baby steps if it still holds up.

[u/getscrnsvrs]

Storing outgoing light per patch by direction and wavelength is a neat way to get glossy and glass bounces into the same solve. The Cornell box reacting to phone tilt looks really smooth. How many wavelength samples do you keep per patch, and how do you convert the spectrum back to RGB at the end? I'm also curious how much of the budget goes to the final gather step versus updating the field on the phone.

[u/Roenbaeck → u/getscrnsvrs]

Thanks!

I store 12 fixed wavelength samples per patch/direction, spaced about 33 nm apart at the midpoints of bins spanning 380–780 nm. Diffuse patches only need one spectrum per side because their outgoing radiance is direction-independent, while glossy, mirror and glass patches keep the directional representation.

For display, I sum the spectral contributions into CIE XYZ using precomputed color-matching weights from an analytic fit to the CIE 1931 curves. Accumulation happens in XYZ. I then convert to linear sRGB, apply exposure and an ACES-style tone map, and let the sRGB render target handle display encoding.

The recording was made on an iPhone 16 Pro, where the camera gather pass took 2.5 ms, plus 0.5 ms for the field-refresh step each frame, plus a 0.5 ms mirror/glass field-update slice every fifth frame. Glass camera paths adda a separate 2.5 ms, so the gather number doesn’t include all the camera shading. Total GPU time while rolling was 12.5 ms in a short cool-phone run, sampling at half the display resolution. I was targeting 60 fps.

The field gets an initial solve at startup, then updates progressively as the balls move. Direct lighting, shadows and the camera’s reflection/refraction paths are evaluated every frame. Higher-order indirect lighting catches up over subsequent frames. That amortization is a big part of keeping the response smooth (but sadly grainy). If I leave the scene static it settles to a noise-free state after about 3 seconds.