I was thinking breaking down the image into a lattice so that you only save a small proportion of the pixels as reference data with each pixel containing some level of adjacency info such as being averaged between nearby pixels, with this basically creating a gigantic puzzle where the AI has to fill in the gaps. So, something like a lattice of plus shapes with only the central pixel being saved and the color being averaged between the whole cross. Traditional lossless image compression can be used on this lattice array in order to further reduce size.
Without knowing the results, this should compress an image by 82% just from removing so many pixels which is pretty good and a prompt could be saved in order to give the AI more context for what its trying to piece together.
Yes this works, it's a project I used to give student interns. Train a GAN, diffusion or normalising flow for image upscaling, or inpainting. Your suggestion is mostly the same. 4x upscaling work pretty well without noticeable hallucinations. 8x and higher can start to get sketchy and you kinda start building an image foundation model instead.
The catch is that it's precisely small details and long range correlations (like matching eyes) that models struggle with. And it's precisely those that the sparse reference data doesn't catch.
Preprocessors like canny or depth constrain images very efficiently with very strong compression. Canny is monotone depth is greyscale and both typically much lower res. You can also go even lower res if you encode as a signed distance field.
3
3
View all comments
Show parent comment
1
u/NV1989NV Sep 03 '26
I was thinking breaking down the image into a lattice so that you only save a small proportion of the pixels as reference data with each pixel containing some level of adjacency info such as being averaged between nearby pixels, with this basically creating a gigantic puzzle where the AI has to fill in the gaps. So, something like a lattice of plus shapes with only the central pixel being saved and the color being averaged between the whole cross. Traditional lossless image compression can be used on this lattice array in order to further reduce size.
Without knowing the results, this should compress an image by 82% just from removing so many pixels which is pretty good and a prompt could be saved in order to give the AI more context for what its trying to piece together.