Libcamera – A complex camera support library for Linux, Android, and ChromeOS

Letting companies protect their intellectual property benefits absolutely nobody but them. They don't have our best interests in mind, only their own. The ideal solution is to reverse engineer their hardware's interface and produce a competing implementation which will no doubt be better than their minimum viable product. My laptop has a backlit keyboard with RGB LEDs and I managed to figure out how it works and make my own program to replace the manufacturer's. I'm so happy I don't have to boot into Windows just to use their slow software anymore.

They don't even have to give us source code. Documentation would eliminate the need to actually reverse engineer the hardware. They could just release that data and all interested parties would start hacking. The problem would take care of itself at no cost to them.

the point is to try to enact change in corporate behavior with regards to consumer devices.

>The alternative is getting nothing.

goes both ways.

If I don't buy a product that uses binary blobs all over the place, they don't sell the product to me.

Supporting products that support DRM systems natively does a lot to support the underlying DRM system, too. It sets a native expectation of DRM among developers, fostering the use.

And worst of all, I can't try to fix blobs without labor that is so intensive that it renders the very idea moot.

Wikipedia faced a similar issue when considering using https everywhere. Some countries were blocking certain pages on Wikipedia and https would break that so it was uncertain if they would just block the whole of Wikipedia now. As it turns out they mostly did not block the whole website and those selected pages became unblocked because the alternative was given by the all or nothing approach was too far.

I have played with ffmpeg, which is not suitable for mobile, unless heavily modified. The VideoLAN folks seem to have gotten VLC working for mobile, but it's a real pig.

In order to capture an image, most image sensors are triggered. Some sensors will operate in "free-run" mode, but in reality something is sending a capture signal to the camera. In sensors where you have very low level control over things, usually what that does is clear the pixels (dump the accumulated charge) and start exposing. Then, when the exposure is complete, the camera makes the data available for read-out (for a phone sensor analogue-digital conversion is likely done in circuitry on the sensor, not in the phone).

That trigger is the important bit. Using triggering you can synchronise multiple cameras (e.g. the sensors in multi-camera phones are almost certainly co-triggered to avoid ghosting), illumination units, etc. In fact many camera chips can also be set to output flash signals when they're exposing, this eliminates latency in sending separate "capture" and "flash" signals. When I've deployed machine vision systems, often we would use an external system to trigger everything (e.g. a microcontroller). Interfacing hardware and software triggering is a nightmare.

Then you've got whatever low level interface is actually communicating to the camera. On a phone, that's probably MIPI/CSI-2. Modern SoCs have dedicated (hardware) camera interfaces (the ISP) that are designed specifically to load images from camera sensors and do some minimal processing. You need kernel-level drivers for whatever your SoC is. These drivers will (in theory) let you set up arbitrary image sensors with whatever resolution/timing they require. But the key is you need two drivers: one to control the ISP and another to send commands to the camera sensor. A lot of the time (cough Broadcom) you only get a blob for the ISP and you almost never get full datasheets for image sensors without being a big customer.

Rockchip have an open source driver for their ISP: http://opensource.rock-chips.com/wiki_Rockchip-isp1

See Rockchip again for an open-source example of an actual sensor driver.

https://github.com/rockchip-linux/kernel/blob/release-4.4/dr...

Want to make an educated guess why libcamera supports the [Rockchip] RK3399? They make it comparatively easy.

In the end V4L acts as a mediator between this low level interface and other things which want to talk to the camera. Rockchip have provided what amounts to a plugin for V4L. Then on top of V4L you have libraries like OpenCV which give you nice interfaces like `imread` to capture images and do stuff with them. To answer your question you probably need to be more specific. The performance of what V4L does for a specific ISP/sensor combination very much depends on those underlying drivers.

If you can afford it, a safe strategy is to leave the sensor in free-running mode in a separate thread that's dedicated to capturing images. Then in your application you pick off the most recent frame and do stuff with it in parallel. For a 30fps camera you therefore have 30ms or so to do your stuff before the next frame comes in.

I've had some really relevant experience, here, myself. I won't go into detail (N of the D-A).

> it will be read and buffered in hardware and set to the system RAM through DMA.

Moving memory, even on the fastest processor, is always a cost. This is especially true, when you look at the pixel count of most cameras (not just DSLRs), these days. Also, you don't just get the memory at once in a big multi-megabyte dump. The sensor needs to be polled. Each cell is queried for its charge state in a linear fashion. There may be ways to access sections in parallel, but I haven't encountered it (It would make sense -I have tended to be on the software side of things, leaving the hardware to the electrical engineering wizards). This is usually dealt with by an ASIC connected directly to the sensor.

If you want to have a "software-only" camera (as opposed to one with a dedicated SoC), then you are looking at some really heavy-duty challenges.

The idea of a "software-only" camera, is to completely do away with the dedicated SoC, or replace it with one that can be re-flashed.

Not so simple.

Computational Photography is something that is necessary for consumer-grade cameras, and is increasingly becoming relevant to pros, as well. It takes some real horsepower.

The newer iPhones have awesome cameras. I suspect that some of the higher-end Androids are just as good, or better. If so, I will bet that you'll find a custom ASIC in the woodpile.

Apple has custom ASICs for image handling, as well as CP. They have an ML chip built into their phones.

I am confident that these challenges will be overcome (there's money to be made, and that's always a great driver), but the challenges are real. There are some very, very smart folks that have beat their brains out on this.

That's precisely why you need good image processing software that works automatically. Nearly everybody wants to take great photos with their phone, but few people know how to configure everything themselves, and even those who do probably don't want to fiddle with every shot.

> Couldn't they just find another niche?

Well, okay, what problem would you rather see them tackle?

It's because people use that camera to record memories that they cherish. Just ask any new parent what happens to their phone's camera roll once number 1 pops out. Or ask young people to show you where they went on their last vacation: many times it'll be a bunch of cool shots taken on a phone.

These memories are priceless and having to record them with a device that is open but has the photo quality of a 5 year old device is not an attractive trade-off.

We already have the hard data: almost without exception, people who use phones want their phones to take nice pictures.

For reasoning about camera software, it is entirely reasonable to assume that “the segment of society that is enthused about photography” is basically the exact same segment as “people who have phones”.

If you don’t believe me, show me a popular smartphone that doesn’t have a camera. :)

Curious, can you name an OS that does a better job compared to android 10? (And maybe a TLDR comparison?)

Huh...? This couldn't be further from the truth. They may not be terribly picky about the technical details about them, but actual human beings take photos... all the damned time.

> The trick to solve the camera problem is just buying a good camera that isn't your phone.

Coupled with your other view on this, it's not worth discussing how ridiculously inconvenient that would be for what people take photos for: social media.

This may not reflect your own usage pattern, but it sure as hell is what real people do with their phones. Taking pictures of their casseroles and making duck lips with their friends.