Camera interface is not via USB on Raspberry Pi. There's a lot of redundancy in high frame rate, so the best way to improve memory footprint is compression.
So you literally do not have the memory bandwidth to do compression while also capturing a good 640x64x32/660 Video.
If you have an RPi4, you get LPDDR4, the memory bandwidth now allows for compression, IF the CPU can handle that throughput (which it likely can't since the RPi4 chokes on such large bandwidths easily).
But you dont even need fancy compression, at those framerates things change really slowly so simple delta encoding on raw data will do wonders.
last time I tried USB 2.0 hdd on pee3 did 35MB/s on the dot
Instead, one needs to use a GPU, DSP, or other dedicated hardware to process/compress the data into a common video or image format.
The first thing of note to me in [1] is the following:
> Digital video cameras perform a process called demosaicking before compression, which normally is lossy. While capturing images, digital video camera, uses only one of the three primary colors (red, green, blue) at each pixel and the remaining colors are interpolated through color demosaicking to reconstruct the full color image. A demosaicking algorithm is a digital image process used to reconstruct a full color image from the incomplete color samples output from an image sensor overlaid with a color filter array (CFA). The most commonly used CFA is Bayer Filter, where the output is an array of pixel values, each indicating a raw intensity of one of the three filter colors. The process of demosaicking, followed by lossy compression is irreversible and the video cannot be improved later on. Further, this process increases complexity, reduces compression ratio and burdens the camera I/O bandwidth.
> [...] if a lossless compression is performed first in the camera itself before demosaicking, then a sophisticated codec can be designed which is needed in applications where high visual quality has paramount importance. This motivated the use of lossless-compression first followed by demosaicking in medical videos where even the slightest degradation result is catastrophic situations. The algorithm uses a hybrid scheme of inter- and intraframe lossless compression to increase the compression and quality of the medical video data. The working of the encoding algorithm is briefed below and is given pictorially in Figure 1.
Skimming through the data sheet, I see a few things:
- Among the features on page 1 it mentions a 10-bit ADC, and “R, G, B primary color pigment mosaic filters on chip”.
- On page 10, there is a block diagram, figure 1.
- Starting at page 52 and throughout various pages of the remainder of the data sheet, there are figures that indicate to me that the output of the IMX 219 consists of mosaic pixel data. So if I am understanding correctly, demosaicing happens outside of the IMX 219.
So I am wondering, with the v2 camera module as a whole, is demosaicing performed on the module itself or in software on the Raspberry Pi? And if it happens on the module itself, can it optionally be disabled so that you can get the mosaic pixel array data?
Since the ADC is 10-bit, each pixel in the pixel array is presumably represented by 10 bits. The data sheet might say, I only skimmed through it on the initial read.
If so, then 640x64 pixels * 10 bits/pixel * 660 FPS ~= 270 Mbps ~= 34 MBps.
A benchmark from 2018 [4] puts the Raspberry Pi model 3B+ at being capable of writing to the SD card at about 22 MBps.
So if we get the mosaic pixel array data, the most simple and naive thing we could do would be to “crop” the image to say 360 pixels wide in memory by skipping past the 140 first pixels of each line and copying the 360 next pixels of each of the 64 lines for a few frames, writing batches of 360x64 pixels of cropped frame data to a single file on the SD card.
If the mosaic pixel array is what we are given by default, or we can get the camera module to send it anyway, then with discarding data we have a starting point. At this point we are able to record for as long as the capacity of our SD card will allow us. 128 GB SD cards are not terribly expensive so it should be possible to record data for like a couple of minutes. Then we can later post process the data.
And if that is possible, then one can start to get really serious and try to device an efficient lossless compression algorithm specifically for our 640x64 pixels of video frames, and perhaps even optimized for different use-cases like horizontal motion only, vertical motion only, and motion in horizontal and vertical directions at the same time.
[1]: https://pdfs.semanticscholar.org/57ee/90a3e59003ec83c2f52aba...
[2]: https://raw.githubusercontent.com/rellimmot/Sony-IMX219-Rasp...
[3]: https://github.com/techyian/MMALSharp/wiki/Sony-IMX219-Camer...
[4]: https://www.jeffgeerling.com/blog/2018/raspberry-pi-microsd-...
And you can hack the camera by injecting I2C commands to achieve effects impossible with the normal Raspbian camera software, like taking odd/even frames with different shutter speed: https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=2126...
For reverse engineering camera I2C traffic, the v1 camera ov5647 datasheet is useful to have: https://cdn.sparkfun.com/datasheets/Dev/RaspberryPi/ov5647_f...