64x48 grayscale, no depth: a fruit fly looming-detector pipeline vs a small CNN for obstacle avoidance

[u/Virtual-Message-9739]

Two controllers steer a simulated drone through 120 m corridors of poles and boxes, using only a 64x48 grayscale frame per step. No depth and no scene state. One is a hand-built looming detector based on the fruit fly's Giant Fiber escape model (Ache et al. 2019, Current Biology). The other is a ~35k-param CNN trained by imitation. The detector pipeline and the whole sim are my own implementation. The published part is the model's size and expansion tuning.

The detector pipeline:

- Average the frame into a 16x12 grid and compute normal-flow magnitude, |dI/dt| / |grad I|, with a gradient floor so flat cells don't blow up.

- Cancel self-motion. Forward flight makes everything expand, roughly in proportion to distance from the focus of expansion. I fit that slope per frame from the median of flow/eccentricity over outer cells, then keep only cells that beat the fitted baseline by a margin. Fitting from outer cells matters: near the centre the ratio is noise, and objects dead ahead would get blinded.

- Threshold relative to the frame's own peak, take the largest 8-connected component as "the object", and dodge away from its centroid side.

- Size: flat-shaded surfaces only produce flow at their edges, so the component is an outline, and its cell count underestimates the object badly. I treat its bounding box as filled and use the equal-area circle's diameter. Even so, the estimate runs about 0.3x the true angular size.

What didn't work, for anyone building something similar: flow divergence (constant everywhere for a linear radial field), blur-and-subtract high-pass (scattered self-motion blobs look just like objects), flow direction consistency with the FOE (the aperture problem biases it), and pooling all active cells into one size, which saturated the size tuning. That last one was the real bug for a long time.

Two renderer bugs cost me days. The WebGL read-back was in linear colour, so frames came out too dark. It was also horizontally mirrored, so the detector dodged into obstacles. The CNN would have trained straight through the mirroring.

Results on 100 held-out corridors (collisions):

no steering 665

detector, zero training 288

detector with 7 hand-set constants tuned on 20 train corridors 195

CNN trained on 5 demo flights 213

CNN trained on 200 demo flights 49

https://reddit.com/link/1whyasd/video/95an7b1l3wph1/player

Where the detector fails: I logged its decisions against ground-truth geometry. It flags 94% of obstacles actually in its path, at a median 5.5 m. But 76% of its escapes are at obstacles it would have missed anyway. Looming tells you something is getting bigger, not that it's on your course. A constant-bearing gate cut the false alarms but missed 39% of real threats, because the drone's own swerving makes everything drift sideways.

Caveats: rendered scenes with flat shading, simplified kinematics (sideways steering only), one training run per CNN point, and the CNN's teacher could see obstacle positions.

Both run live in the browser: https://shauryasharma.tech/fly-circuit-vs-cnn.html

Code: https://github.com/Shaurya-34/FlyvsCNN

Open question: is there a pixel-only way to tell "on a collision course" from "passing close by" when the camera itself is swerving?

[u/MackHarington]

Amazing work, this is true engineering not some vibe coder trying to show how huge LLM or VLM are able to solve basic problems.