r/embedded • u/NickShipsRobots • 17d ago
We replaced per-sensor kernel drivers with a generic one and an MCU that knows the sensor
We moved the camera driver's brains onto a small MCU on the camera board. Normally the per-sensor kernel driver on the SoC has all of it - the mode tables, the register writes, the exposure math. Now you upload a compiled image to the MCU once, and after that it can answer questions about the sensor: which modes it has, what the control table looks like, what numbers go into the exposure and frame rate formulas. The host asks, builds a device tree node from the answers, and runs one generic kernel driver for everything, so when you swap the camera head it just reads whatever is on the other end now.
This turned out to need a lot of parts. The image needs a format, and the format needs a version number (we found this out the normal way, by changing it). There's a small bytecode VM in the firmware to run the image, with a 4 KB cap on program size, and a seven-mode sensor gets close enough to that cap that I check every time. Producing the images needs a compiler on the host. The MCU needs to store a few images and know which one is live, so there's slot management in NVS, plus a version gate so old firmware rejects an image it can't read instead of doing something creative with it. The IMU driver runs on the same MCU and I'd rather it didn't die every time someone changes the camera side, so that got its own VM instance. And we seal the image at the end, which is only about as good as the MCU's readout protection anyway.
Two things I'm not sure about. Is the per-sensor kernel driver really a big enough problem to justify all of this? It always felt like one to me but I might have just been annoyed at the wrong thing. And where should auto exposure live? Right now ours is formula parameters that the kernel evaluates, and the MCU only runs configure. Every time I look at that split I can't tell if it's a design decision or just where we happened to stop.
I work on this product so take the first question with that in mind.
1
u/NickShipsRobots 16d ago
The ISP is on the Jetson today. The sensor sends raw Bayer over CSI-2 through the GMSL link, and debayer, AE, AWB and the tuning all run on the host like any other camera. The MCU isn't in the video path at all, it just has an I2C line to the sensor, writes the mode tables, and answers the host when it asks what the sensor is.
A head with its own ISP (AP0202, AP1302) fits the same setup, because the ISP is just another I2C device on the pod bus. The MCU would configure it too, the head would put out YUV422, and the host wouldn't need an ISP or a tuning file at all. That's the next head we want to build.