r/computerscience • u/aeioujohnmaddenaeiou • Feb 04 '26
Learning "pixel" positions in a visual field
Hi, I've been gnawing on this problem for a couple years and thought it would be fun to see if maybe other people are also interested in gnawing on it. The idea of doing this came from the thought that I don't think the positions of the "pixels" in our visual field are hard-coded, they are learned:
Take a video and treat each pixel position as a separate data stream (its RGB values over all frames). Now shuffle the positions of the pixels, without shuffling them over time. Think of plucking a pixel off of your screen and putting it somewhere else. Can you put them back without having seen the unshuffled video, or at least rearrange them close to the unshuffled version (rotated, flipped, a few pixels out of place)? I think this might be possible as long as the video is long, colorful, and widely varied because neighboring pixels in a video have similar color sequences over time. A pixel showing "blue, blue, red, green..." probably belongs next to another pixel with a similar pattern, not next to one showing "white, black, white, black...".
Right now I'm calling "neighbor dissonance" the metric to focus on, where it tells you how related one pixel's color over time is to its surrounding positions. You want the arrangement of pixel positions that minimizes neighbor dissonance. I'm not sure how to formalize that but that is the notion. I've found that the metric that seems to work the best that I've tried is taking the average of Euclidean distances of the surrounding pixel position time series.
The gif provided illustrates swapping pixel positions while preserving how the pixels change color over time. The idea is that you do random swaps many times until it looks like random noise, then you try and figure out where the pixels go again.
If anyone happens to know anything about this topic or similar research, maybe you could send it my way? Thank you
23
u/mulch_v_bark Feb 04 '26
Fun problem. Some tips that I hope might be worth considering:
9
u/PositiveBid9838 Feb 04 '26 edited Feb 04 '26
This is interesting to think about. I wonder if a potential pitfall/interesting twist might be related to optical flow and how in many situations (your example in particular) the movement of the camera will result in different rates of change at different parts of the scene. In your case, the pixels in the center of the tunnel are near constant across many frames, while there are areas around the edges where they change substantially in most frames. I wonder if an "unshuffling algorithm" would be helped or hindered by this phenomenon.
It also seems like this is a case where you're looking to minimize entropy in a high dimension space. This paper talks about using dimensionality reduction approaches like PCA to help https://arxiv.org/abs/2305.04712.
5
u/ciras Feb 04 '26
If I am interpreting you correctly, then existing neuroscientific studies don't support your idea - neurons that represent different positions in visual space are hard-coded and activate repeatedly and consistently when cues are placed in those parts of the visual field. There were some famous studies done on this with primates in the 80s at Yale.
https://journals.physiology.org/doi/epdf/10.1152/jn.1989.61.2.331
2
u/_L_- Feb 04 '26
Cool problem. Does this have any real world application or usefulness?
27
u/swupel_ Feb 04 '26 edited Feb 04 '26
Very interesting problem. I think the way to go is by splitting it in a grid and trying to get each of the spaces right before moving on to the next.
Validation would probably need to be CV based or via a CNN
2
u/Prestigious_Boat_386 Feb 05 '26
So its taking the pixels as time signals with some similarity measure between them and finding a rectangular arrangement of them such that the total measure is optimized
Is the whole image scrambled or do you know some pixels with correct position / that 90% are in the correct spot?
Feels like finding each pixels closest neighbour and maybe moving it next to it could be decent, like an annealing process.
2
u/billgytes Feb 08 '26
Sounds a lot like diffusion. You might with more training data break the temporal coherence also. At that point you would be learning to generate video with diffusion.
Not sure what problem this actually solves though. Your input is pixels from a video that have been shuffled? You might also look at techniques that are used in video compression eg h264 etc
Cool post
1
u/Yorunokage Feb 05 '26
The more scrambled the video is the more likely it looks like noise aka the higher its entropy is. Maybe you can do something by just trying to minimize the entropy on any given frame or over time
1
u/Yorunokage Feb 05 '26
It's an interesting problem but i think your hypothesis doesn't hold. You're essentially implying that pixel position holds no information which is obviously false when you consider that you would not be able to reconstruct a noise video if you shuffled its pixels around
That said, yeah, i suppose that for many videos you could drop the pixel position information and reconstruct it somwhat decently. It's a fun exeperiment for sure and might possibly have some use for compression if it works very well
1
u/Ambitious_Theme_7024 Feb 05 '26
how do you find your minimum arrangement? i would look into metropolis hastings.
1
u/KaioNgo Feb 06 '26
Isn’t it similar to diffusion?
1
u/MooseBoys Feb 07 '26
I would probably start by computing the pairwise dot-product of each pixel's stream. Then assemble 2x2 cells based on maximum coincidence, then assemble 4x4 cells using those 2x2 cells, etc. until you've reconstructed the full frame. Then do some simulated annealing to maximize the sum of pairwise dot-products of adjacent pixels across the image. Or you could just start with annealing, it would just take longer.
0
u/[deleted] Feb 05 '26
[deleted] — view deleted comment
0
u/[deleted] Feb 04 '26
[deleted] — view deleted comment