Cameras and Open Source Smartphones: It’s Complicated
puri.sm
puri.sm
The problem of camera diversity is not limited to open source either, because a similar infrastructure to handle all the different cases must be replicated by closed drivers as well. I don't know about Macs, but the Surface laptop is a Windows beast.
In reality, UVC is not suitable for a phone, so we can't leverage that. There are some camera drivers in the kernel already, but not necessarily ones that we could buy or meeting our expectations. Even if there were, that still leaves us with the problem of connecting the cameras to applications in a standard way, so we can't really avoid working on libcamera.
I think it's going to be a long way to get there, but also the openness of the drivers will let us find our own strengths (I have high hopes for super-high FPS recording).
That would be very cool. Google’s phone does 4K HDR stabilized video at 60 fps. Their slo-mo is 240 fps but I don’t know what resolution that would be.
So why not fork it and make libcamera2 (or whatever) and concentrate solely on Purism’s needs?
My gut instinct would be that unless you ruthlessly narrow the scope of the project, progress will be too slow.
For likes of Librem and Pinephone, the highlevel controls either don't exist on HW level, or they exist there, but are not exposed on the video device itself, but on various v4l2 subdevices that form the video pipeline.
One way to support all the already existing apps would be to implement what they already expect. (see above) That is, to be able to control the video device by usual means they already posess. Instead of extending all the apps to use libcamera, and leaving the rest behind, we could simply proxy the video controls from ioctls where all apps expect them to be to some userspace daemon, that would then configure the complex HW specific media pipeline behind the scenes (basically all the media system subdevices for sensors, sensor interfaces, ISP, etc.).
In other words to implement what's implemented by some USB microcontroller in UVC webcams in some userspace daemon, but keep the existing userspace interface expectations for simple camera usage.
This is kinda orthogonal to the libcamera effort, I guess. Just wanted to say that there already is a standard. :)
It could be a viable way to get the basic use case of video streaming where special controls are not needed. It's worth considering, although then it makes sense to leverage work already in libcamera to implement the extra layer.
Many SLRs are already well supported, though the open source stuff doesn't focus on low latency conversions, which is needed for viewfinders, or focus control, etc.
If I have to rewrite stuff for low latency, I'd rather start it as an independent library so that other projects can reuse the code.
I did a side-by-side comparison of a Micro Four Thirds camera (4/3" sensor) and an iPhone SE (1/3" sensor) and the performance was... pretty much the same.
And I'm not talking about some ML interpolation wizardry or automatic face beautification; I was photographing barcodes and testing how many were readable in the resulting images - hardly something Apple would have specially optimised for.
The iPhone has a much smaller sensor, a much smaller lens, costs less, and manages to pack in a bunch of non-camera features. To be competitive in the modern cell phone market your camera has to be straight up magic.
But when I say the performance is good, I'm not just declaring the images good because the portraits have simulated bokeh, or face-detecting autoexposure, or image stabilisation, or tasteful HDR, or a beauty mode that airbrushes out blemishes and makes photos of sunsets really pop.
Even in applications where none of those features come into play, the iPhone, can still go toe-to-toe with cameras with much larger sensors.
Did you compare raw sensor output, or post-processed?
Big sensors capture more light and have more bokeh. With enough light, the first doesn't matter, and bokeh is not a thing for QR codes.
If you didn't have enough light, then it's probably the question of how denoising was done, and what details have been guessed by fancy algorithms. Geometrical shapes are easy to guess, but when I look at pictures of landscapes, they typically devolve into a painting after 1000×1000px if taken on a phone.
https://semiconductor.samsung.com/image-sensor/isocell-plug-... EIS video: https://images.samsung.com/is/content/samsung/p5/semiconduct...
Unfortunately I never got one and it now looks to be unavailable. I don't know if Samsung would have had interest in Open Source efforts to support the modules.
VCX objective camera evaluation links: https://www.dhd.com.tw/wp-content/uploads/2020/11/VCX_Whitep... https://vcx-forum.org/
I assume most people have no visibility of the effort of hundreds of engineers who work on behalf of the big smartphone companies tuning for what happens in the image pipeline between camera/raw Bayer data to what your app receives.
There is a reason the quality of pictures in phones has got so good - lots of tuning, quite a bit of magic, as well as software and hardware co-optimization.
The low end of professional-ish cameras is great in lots of ways, but they all have annoying trade-offs. To be fair I'm sure any janky POS I built would too.
Reason enough to support purism.
edit:
This would also resolve the dilemma of having complex camera drivers either in kernel or in a library. It could be in userspace while the kernel provides a small kernel module to make it work (like v4l2-loopback, or not unlike FUSE for filesystems).
So the interface is the same, it's just split over many device and platform specific subdevices, which normal apps can't really be expected to understand.