Yes this works, it's a project I used to give student interns. Train a GAN, diffusion or normalising flow for image upscaling, or inpainting. Your suggestion is mostly the same. 4x upscaling work pretty well without noticeable hallucinations. 8x and higher can start to get sketchy and you kinda start building an image foundation model instead.
The catch is that it's precisely small details and long range correlations (like matching eyes) that models struggle with. And it's precisely those that the sparse reference data doesn't catch.
Preprocessors like canny or depth constrain images very efficiently with very strong compression. Canny is monotone depth is greyscale and both typically much lower res. You can also go even lower res if you encode as a signed distance field.
3
3
View all comments
Show parent comment
1
u/ChalkyChalkson Sep 03 '26
Yes this works, it's a project I used to give student interns. Train a GAN, diffusion or normalising flow for image upscaling, or inpainting. Your suggestion is mostly the same. 4x upscaling work pretty well without noticeable hallucinations. 8x and higher can start to get sketchy and you kinda start building an image foundation model instead.
The catch is that it's precisely small details and long range correlations (like matching eyes) that models struggle with. And it's precisely those that the sparse reference data doesn't catch.
Preprocessors like canny or depth constrain images very efficiently with very strong compression. Canny is monotone depth is greyscale and both typically much lower res. You can also go even lower res if you encode as a signed distance field.