"I worked on a embedded hardware application using Linux and "cameras". We (like many people) were using sensors from OmniVision[0] and there was a dedicated contractor who's entire area of focus (and life's work) was getting photons from the sensor of the camera into something usable by software. In our case that was an unholy combination of vendor software, custom code derived from very obscure NDA-only datasheets, and a bunch of custom plugins for gstreamer that could actually make use of the sensor, interfaces, and hardware encoding silicon.

It was one of the more eye-opening "WOW, so that's how the sausage is made" experiences of my life."

This matches my experience, bringing up a new MIPI camera on any of the big name SoCs is a nightmare. Actually I've seen a lot of fails with gstreamer, where for one reason or other gstreamer doesn't work, so the software just uses the image sensor driver directly (usually V4L2). But for many SoCs, this means that you get no IPU support, since it's only available as a gstreamer plug-in.

(try to make a webcam using gstreamer: ffmpeg works better...)