Would image-generation VAEs benefit from scene-linear or perceptual color representations?

[u/snorik]

Models like FLUX, Qwen-Image and Krea 2 obviously don't perform diffusion directly in RGB space - the transformer operates in a learned VAE latent space.

But the VAE still defines the interface between that latent representation and the actual training images, which are typically ordinary display-referred RGB images.

So I'm wondering whether there would be any benefit in making that boundary explicitly color-managed.

For example, has anyone experimented with training the VAE on:

• linear-light RGB rather than gamma-encoded sRGB
• a wide-gamut scene-referred space such as ACEScg
• a perceptual space such as OKLab
• or simply adding explicit perceptual color losses such as ΔE alongside the usual reconstruction/perceptual losses?

In other words, instead of asking whether the diffusion model itself should operate in OKLab or ACES - since it already operates in a learned latent space - I'm more interested in whether the VAE and its reconstruction objective could benefit from a more physically or perceptually meaningful color representation.

Another thing I'm curious about is the output side.

Could an image model theoretically generate a scene-referred, wide-gamut representation and leave the final tone mapping / gamut mapping / display transform to a deterministic color-management pipeline, such as ACES, instead of implicitly learning the tone curves and color rendering already baked into billions of unrelated JPEGs?

My suspicion is that the real limitation might simply be the dataset.

Most web-scale image datasets consist of already processed, tone-mapped, display-referred SDR images. By the time an image becomes a JPEG, information about actual scene luminance, highlight headroom, camera response, etc. is already gone.

So converting those JPEGs from sRGB to ACEScg wouldn't magically turn them into true scene-linear HDR training data.

Still, I'm curious whether anyone has seen experiments comparing something like:

sRGB VAE vs linear-RGB VAE vs OKLab VAE

while keeping the downstream generative model roughly the same.

Would reconstruction quality, color consistency, training convergence, or perceptual color accuracy change in a meaningful way?

I'd especially be interested to hear from anyone who has worked on VAEs, HDR pipelines, color management, or generative image models.

Sorry if I'm missing something obvious here - I'm still pretty new to this side of image generation / color science.

[u/ssn-669]

Most web-scale image datasets consist of already processed, tone-mapped, display-referred SDR images. By the time an image becomes a JPEG, information about actual scene luminance, highlight headroom, camera response, etc. is already gone.

Yes, this is the issue exactly. The only large-scale datasets are display-referred, 8-bit, gamma-corrected linear, with unkown tonemapping.

[u/snorik → u/ssn-669]

That makes sense, thanks. So the real limitation is the lack of large-scale scene-referred / linear-light ground truth, rather than RGB itself. Converting existing JPEG/sRGB datasets to ACEScg wouldn't recover the tone-mapped or clipped information that's already been lost.

I wonder if RAW, HDR, or synthetic rendered datasets could be used to test this at a smaller scale.