The growing image-processor unpleasantness
lwn.net
lwn.net
Thats suboptimal, because there are lots of bits of research into better ways to do focus. Todays image processors typically just adjust back and forth till they find a frame with more high frequency components. It kinda works, but requires them to overshoot a bit and go back, and doesn't work well for things like mirrors or windows where they may focus on the window rather than whats behind it or vice versa. They also have to adjust the focus slowly because they need to get a new frame and process it to see that the focus needs adjusting further - despite the fact the coils in the camera can move the lenses in milliseconds if commanded to.
Future image focus techniques do things like run machine learning over the image, and say "thats a car, and it looks about 25 yards away, so set focus straight to 25 yards".
Point your phone at your hand, the same distance away as the flowers. Focus it on your hand. Lock the focus (and exposure and white balance - it seems you can never lock one without the other). Now point the phone at the flowers and take the photo.
Expensive phones normally support camera manual focus if you install the right app.
As for the provided user-space software, it's not clear why this is happening. As long as kernel modules keep to the allowed interfaces, (and not the GPL-only internal interfaces) they can be closed source. Is it a case of too much caution from the vendors (after all -- NVidia have closed source drivers)? Is this supposed to be more forward compatible? Is it possible to have a higher performance (zero copy?) user-space exposed abstraction for this kind of video processing?
Good to see that the good old "laptop doesn't go to sleep on linux" problem has never truly gone away.
The HDMI issue is ever present. Intel has lately been dropping the ball significantly with their drivers, but at least it's not Nvidia this time, so that's a change!
But yes, I've had 4 Dell XPSes, all running windows, and all of them have had at least one episode of the laptop cooking itself inside a bag - I pull it out and it's literally burning to the touch, because some stupid event decided to wake it up, despite the lid being shut.
The good thing about Dell is that they ship with linux on certain configurations, so linux installs are very robust with the caveat that you will not see the agressive fan curves or higher tdp limits that you can get on windows.
And it wasn’t even just a recommendation, they declared it would void your warranty to do so! (In at least Australia, such an attempt to disclaim liability is illegal.)
FWIW, I have a spare Windows laptop, just over a year old, whose WiFi does not recover upon waking from sleep about eleven times out of a dozen.
OTOH, I have had both Windows and Linux laptops which have gotten uncomfortably warm when ‘asleep.’ And right now I have a Linux laptop which very rarely hangs on sleep and turns into space heater, not even turning off the monitor but becoming completely unresponsive.
Sleep is apparently a lot trickier a feature than it looks.
On desktop, sometimes there's no video signal after wake up.
The laptop doesn't support sleep at all, it's either connected standby = not sleeping, or hybernate. Windows says none of the proper S1-S3 states are supported by the computer.
He's not kidding. It's hilarious. It could be a driver for anything.
https://chromium-review.googlesource.com/c/chromiumos/third_...
I might be misinterpreting this, I'm not at all a uardware/driver person, if that's even a thing. Can anyone confirm this?
[1]: https://github.com/intel/intel-camera-drivers/tree/master/dr...
Basically the main difference is that with IPU-based stuff the kernel must support not just a "webcam"-like device, but rather must contain support for all the individual components making up the webcam. Think not only actual raw sensor hardware, but even stuff such as controlling the shutter and voice coil motors directly.
The situation is similar if not identical to a smartphone where the OS interfaces directly with the raw camera hardware instead of like in traditional computers where it was all abstracted in the form of a USB UVC webcam. In fact, some of the newer sensor drivers are adapted/stolen from Android kernels.
Many, many laptop models come with IPU3 hardware, which is around 7 years old hardware at this point, and it is still mostly unsupported in Linux. In fact, practically _all_ the support in current Linux comes from the efforts of one guy who has been trying to make the Surface Pro line of devices work under Linux ( https://github.com/linux-surface/linux-surface/wiki/Camera-S... ). I have had some success with these patches (which are now mostly upstream), by manually writing/converting from Android kernels/HALs the required sensor drivers in my laptop, but the image quality was just garbage. I don't even know how much image processing is done software-side on Windows.
Intel has published code, but not bothered to upstream it; Google, who is using this hardware in some ChromeOS devices, only cares about ChromeOS support which does things just too differently to Windows devices (e.g. basic things like enumerating which camera hardware is connected). None of the big OEMs is behaving nicely here.
IPU4 appeared more recently and is basically entirely unsupported. There is zero upstream support and don't bother with the out-of-tree driver since it will be a waste of time (you also need the drivers for the specific hardware sensors used in your laptop + the "pipeline configuration").
IPU6 is probably just a newer iteration of this. I don't know if it will have a bit more developer/manufacturer oomph and thus more Linux support than previous iterations, but even if it does, it's highly likely that all existing laptops with IPU hardware will never have a working webcam in Linux.
It reminds me of a comment [1] I left a while back, and that I randomly found again today:
> Even though significant work is required to keep [old machines like 2012 thinkpads] working, some people prefer to do so. Crucially, they prefer it not because they love tinkering, but simply because the value propositions of 2021/2022 hardware doesn't look better.
I guess the value proposition of modern notebooks is actually degrading, at least in this respect...
I know at least in rPI land this is not a matter of unwillingness but inability: the GPU boot blobs are protected under copyright law by upstream licenses that do not permit distribution of the source (which is not f/oss anyway).
It’s like three vendors removed (GPU firmware author, SoC maker, board maker). I imagine the ISP situation is somewhat similar; Intel probably has licensed some of this code versus making all of it from scratch.
The firmware industry is like an alternate reality where f/oss never happened, it seems. People guard their codez as jealously as the underground blackhat scene.
Code libraries for implementing stuff in hardware (on FPGAs) is literally referred to as “IP”. The mind reels. Totally different culture over there.
https://forums.raspberrypi.com/viewtopic.php?p=16989#p16951
>The GPU contains a very sophisticated ISP - the pipeline of image processing to get some raw camera data to pretty JPG's. This pipeline also needs to be 'tuned' to the particular camera and this can take some times (months for a specialised team to do a really good job) >All the work above needs to be done for each new camera module - and will be done for any camera sold for the Raspi.
Raspi cameras ship with drm https://gist.github.com/marcan/6dde73a9a0c917cd4fc9784a0a73e...
but the actual register writes to drive the ISP are still in the blob, and libcamera is just telling the blob how to tweak every knob in the hardware
it makes no sense why they have opened it up so much, yet still insist on keeping that last bit a secret
Just like GPU's were first dedicated hardware that could only render triangles, but are now fully fledged parallel compute machines which, if asked to, can also render triangles.
Image processors will be the same - and eventually the parallel compute abilities will be merged into a GPU.
They'll be replaced with GPU's that can directly input the gigabits of data that comes from a CCD sensor straight into the GPU RAM, ready to have noise reduction, demosaicing, de-shake, stacking, de-warping, focus, white balance, etc. algorithms applied to it.
The Moore’s Law/Denard Scaling disconnect has created a growing need for dark silicon [1] on chips. Thus we are likely to see even more specialized components rather than fewer. The Apple M1/M2 is a perfect example.
And GPUs and image processors look a lot alike. The largest difference today is on the IO links, but it makes too much sense to link the camera directly to the DRM.
If you use a component too frequently, it isn't dark enough.
I'll trust your statement that GPUs and image processors look a lot alike (I am no expert by any means), but as long as there is a difference an image processor is going to be a candidate for inclusion on the chip. The designers have a transistor budget, thermal constraints, and a prioritized list of components they can put on the chip given available transistors. I'm sure that if they had better alternatives, they'd skip the image processor. But they didn't, which gives you some information on how Intel thinks about image processors.
The real question is where the hell is the open source hardware?
Engineers working for NVIDIA, or camera IP core specialists are not gods. Let's just make our own god damn camera and GPU.
Thanks to nouveau being mediocre at best and lagging 2 or 3 generations behind until it has usable support for a GPU, I always assumed writing a decent GPU driver through reverse engineering must be next to impossible.
So what's up with Asahi? Were there just some gifted Wunderkind working on it? Is the architecture so much simpler? Is there more information available? Iirc it's based on powerVR, so not completely unknown, but then again the nouveau team got some hints from Nvidia in the past, and also they have been working on it for a really long time in comparison. Just really curious.
(“Dissecting the Apple M1 GPU, Part II”)
Nvidia is giving these blobs to Windows users who download their drivers, so why can't they write a script/etc that extracts blobs from a Windows driver binary and let the user download that binary manually?
Where have you read that? According to Asahi Lina's tweet from a few days ago, they are just starting with it:
No. That has indeed happened. It was a work by Alyssa Rosenzweig [0].
If I understand correctly, she implemented the userspace Mesa driver. The kernel driver is being developed by Asahi Lina. Together, they are meant to provide "full open source graphics stack for 3D acceleration on Asahi Linux" [1].
[0] https://rosenzweig.io/blog/asahi-gpu-part-5.html
[1] https://rosenzweig.io/blog/asahi-gpu-part-6.html
Edit: make the quote more accurate, adjust formatting
Or rather: based on the results of other contemporary complex hardware RE efforts, Asahi will never be relevant. Think of _how many GPU hardware_ there have been efforts to RE, and how few have produced any drivers that people would be happy to use on a daily basis. How many people use Nouveau these days? PowerVR SGX (a "GNU high priority project" from a decade ago!)? Lima? Even Panfrost?
In fact, this entire IPU webcam thing is not new. We've had hardware with IPU for at least half a decade and literally _NO_ such device works with Linux, only a couple devices half-work, and very poorly at that.
There are much fewer people working on hardware support on Linux that you may think.
Agreed. I get so angry every time I think about the amount of Verilog and VHDL that will never, ever get released even after the companies responsible for it die and even after many years have elapsed.
Where is the 3Dfx VSA-100 source, for instance? What technical advantage could it give NVIDIA now? They purchased 3Dfx twenty years ago!
Don't get me wrong those peoe are heroes. They're literally sacrificing their life time for a good cause. But like with many such many hero stories, some bureaucrat could have resolved the issue with the stroke of a pen.
What we need is an extreme push against a culture of secrecy and disempowerement of device owners.
Not necessarily. Impose a tax on all products where the manufacturers do not release all documentation even for past devices, that should serve as a pretty decent incentive.
Yes, legally it's a questionably grey area, but to be honest it's time to actually use our market power.
Arguably, refusing to publish literally every single document pertaining to proprietary hardware is not on the same level of obivous malpractice as a genocide, so I think you could have proposed a milder example to argue your point.
I would never dream to equate proprietary documentation to Nazi war crimes. That's without question. I just used this because, like so often, in questions about these war crimes, there is no real argument to be made for "the other side" so it clearly shows the line of reasoning for breaking with a legal convention in specific cases. You said it yourself: "it's something so intrinsically evil that you can't rely on pure formality." Which is exactly the point. Protecting the perpetrator just because there's also a value in relying on their actions being legal at the time just doesn't outweigh the cost of letting them get away with such monstrosities.
A more recent and less evil example would actually be the Cum-Ex scandal. One question was whether money would have to be given back, given that what happened might be technically-legal but blatantly, and expressly against the spirit/intention of the law. But that whole thing is still being fought out so the conclusion is less clear cut.
It’s real world training for reverse engineers. Having worked in that field myself I can tell you I was never surrounded by so many smart, talented and exceptional people as when I worked with other reverse engineers (especially in the video game world).
RE is a necessity, and it paves the way for some incredible talent to shine and practice.
A global optimum would include some other way for these legends in the making to train.
Completely non-existent? Either a laptop comes with windows or you get zero support. See every chromebook ever.
Unfortunately, the FOSS world isn't always able to cater to professional and prosumer crowds.
If it is absolutely critical to my work that a piece of hardware has feature X, then I'm going to buy it regardless.
I'd choose a FOSS-compatible solution if it were available, but this hasn't been the case many times in my career. (Yes, I can provide concrete examples if necessary.)
I wish I was knowledgeable enough to tackle more complex hardware like webcams. Perhaps one day I will be.
https://lkml.org/lkml/2020/9/17/363
People who started the job on Surface webcam hardware didn't even know C to begin with ;)
I was able to intercept and reverse engineer much of my laptop's USB features. I hit a wall when I tried to figure out the ACPI stuff. There's some fan control functionality in the Windows app that's missing from my software but the fans are terrible and always at 100% anyway so at some point I decided to just let it be.
It seems to be that making such documentation available should be the core demand on hardware manufacturers, not that they necessarily open source their own software/firmware.
That would work only if companies could be compelled to release complete documentation. E.g., Microsoft even corrupted[1][2] the ISO standardization process to force through standardization of their OOXML (to take the wind out of worldwide efforts around standardization on the existing ISO standard, ODF) even though the OOXML Microsoft shipped _after_ standardization could not be implemented from the specification in the standard [2].
Forcing companies to provide source for a fully functional driver under a suitable license would prevent this sort of abuse from bad actors.
[1] http://www.groklaw.net/article.php?story=20070312083134403
However if the nature of a device is such that a driver for it requires the implementation of non-trivial algorithms then I wouldn't support government action to force companies to open source such code.
I think standardization processes are a rather different topic than hardware, but I'm wondering, did the Microsoft shenanigans you mentioned actually prevent people from developing compatible software ? My impression is that the compatibility issues that people had with MS file formats in the 90's have largely been resolved in practice. Is that not the case ?
I don't follow the MS world, but I do recall complaints that OOXML as built by MS included binary blobs (straight out of their old, undocumented, doc and xls formats) wrapped in XML. And, folks using MS software at old job sometimes complained that when I edited an MS document in libre office, "I broke it." Thankfully my group standardized on plain text to make version control more useful, so it only came up when dealing with folks outside the group.
You could maybe technically argue things like denoising algorithms / foreground/background detection might be more secret as there might be competitive advantages there somewhere...
We (like many people) were using sensors from OmniVision[0] and there was a dedicated contractor who's entire area of focus (and life's work) was getting photons from the sensor of the camera into something usable by software. In our case that was an unholy combination of vendor software, custom code derived from very obscure NDA-only datasheets, and a bunch of custom plugins for gstreamer that could actually make use of the sensor, interfaces, and hardware encoding silicon.
It was one of the more eye-opening "WOW, so that's how the sausage is made" experiences of my life.
[0] - https://www.ovt.com/
Your code delivery could also contain parts that are licensed from other companies, limiting your options with regards to licensing.
One way to limit the problem is to put most of your "secret sauce" code onto one or more sub-processors embedded in the imaging hardware. The register bank of your hardware would only be accessible to these sub-processors, meaning that you don't have to publish your register definitions. Then you can deliver the sub-processor microcode as a binary blob (independent of the host OS) and let the kernel module itself be open-source. This approach is taken by at least some vendors.
The "subscriber link" mechanism allows an LWN.net subscriber to generate a special URL for a subscription-only article. That URL can then be given to others, who will be able to access the article regardless of whether they are subscribed. This feature is made available as a service to LWN subscribers, and in the hope that they will use it to spread the word about their favorite LWN articles.
If this feature is abused, it will hurt LWN's subscription revenues and defeat the whole point. Subscriber links may go away if that comes about.