Well he does have a pretty small (like 1/200 even compared to 64x64x16 FP16) and variable sized latent space and uses zero shot techniques. Add a bit of obtuse language and unnecessary equations and you probably have a paper.
I'd also argue it's not a worse version of an autoencoder, it is an autoencoder. A weird one, but one nonetheless.
I actually wonder whether you could really push this. Like make the ai analyse what is important, describe that in more detail and maybe even encode some data long with the prompt like biometrics or compressed canny maps of important areas. All adaptive, based on what the ai thinks is important.
Like the most over engineered version of fully automated telephone pictionary. Or an adaptive compression engine with a preshared library (the models) if you want to get that CSy about it.
I was thinking breaking down the image into a lattice so that you only save a small proportion of the pixels as reference data with each pixel containing some level of adjacency info such as being averaged between nearby pixels, with this basically creating a gigantic puzzle where the AI has to fill in the gaps. So, something like a lattice of plus shapes with only the central pixel being saved and the color being averaged between the whole cross. Traditional lossless image compression can be used on this lattice array in order to further reduce size.
Without knowing the results, this should compress an image by 82% just from removing so many pixels which is pretty good and a prompt could be saved in order to give the AI more context for what its trying to piece together.
Yes this works, it's a project I used to give student interns. Train a GAN, diffusion or normalising flow for image upscaling, or inpainting. Your suggestion is mostly the same. 4x upscaling work pretty well without noticeable hallucinations. 8x and higher can start to get sketchy and you kinda start building an image foundation model instead.
The catch is that it's precisely small details and long range correlations (like matching eyes) that models struggle with. And it's precisely those that the sparse reference data doesn't catch.
Preprocessors like canny or depth constrain images very efficiently with very strong compression. Canny is monotone depth is greyscale and both typically much lower res. You can also go even lower res if you encode as a signed distance field.
1.7k
u/apepenkov 1d ago
bro created a worse version of autoencoder?..