Back to Android for the time being
proxy.vulpes.one
proxy.vulpes.one
I've spent few weeks of my (spare) time improving a Sony imx219 driver for a Rockchip rk3566 cpu that is used in few pine devices I own. I needed high fps video (I managed to get 120fps 720p - I think I could push it further to 140 perhaps, but this was OK for now) and as low latency as I could get. I wanted <20ms as I could encode 720p h264 frames in 7ms,unfortunately I got stuck with ISP (image signal processor) adding 40ms of latency even when in pass through mode.
The modern ISP is a "black box" even with source code for the drivers. In addition every single ISP has its own proprietary format for those profiles. There are OS projects that try to make universal camera sensor profiles that work with multiple ISPs (libcamera for one), but it is still early days.
What we need is a truly programmable/flexible isp. As it stands the only way to get sub 40ms of latency from a camera for a hobbyist is to use an fpga...
Over time the open source crowd implemented the state of the art image processing techniques once, then pointed them at reverse engineered backend drivers for each model.
The result was better than what the manufacturers (especially HP, but also canon, etc) could do.
I'm hoping history repeats itself with these camera sensors.
It does not make the calibration (or preparation of profiles for various light conditions) easy though. There's a need for a bunch of HW tooling that is somewhat expensive for a hobbyist. And there's a lot of work and algorithm design that needs to be done outside of ISP for controling the autofocus, autoexpsoure and detection of type lighting in the scene and profile switching. None of this is particularly trivial.
I worked with rk3566, but I believe rk3399 has the earlier version of the exact same isp. The version in rk3566 is 1.1. There are also versions 1.0 and just 1. All are documented in TRMs in the same way.
Edit: BTW, if you need the tooling (Rockchips isp profile development tool v2.1 and the manual for it) please let me know and I'll send it to you.
It's not usable for RK3399 version of the ISP. BTW the same/very similar ISP (to RK3399) is used in NXP SoC, and it has documentation there, too. (NXP docs tend to be better ;))
If you compare with some other manufacturer that gives us absolutely no docs, then yes having register level docs and user side source is really nice. But if you want to do something unconventional (like switch the whole damn thing off and bypass its latency) not knowing how it works internally is still a black box. For example there is an irq that fires when the first component of the isp(MI-memory interface) sees a vsync signal for a frame. We know it exists. We know how to enable/disable that irq. We can write code that does stuff when it fires. But what we don't know is is this vsync signal fired truly at the input of the ISP? It's output? Or perhaps between some specific components? It turns out that vsync fires not when the isp sees the very first vsync what the location of the registers suggests, but after all the processing is done at the output buffer (40ms later usually), but one wouldn't be able to tell without a lot of profiling. This amongst other things is what I call a black box.
However, compare it with let's say a raspberry pi, and at register level the documentation is much better (for the isp).
There are bunch of interrupt statuses around frame capture: RIS_FRAME_IN, RIS_FRAME, RIS_V_START.
Not sure why you'd decide to use V_START for deciding when the frame is in memory. VSYNC start is sent at the start of a frame and it's documented as just a interrupt that is directly triggered from the MIPI CSI sync signal in the datasheet.
quote: "The incoming hsync and vsync control signals from the camera are connected to interrupt logic. It is possible to trigger on these signals for an event triggered configuration during processing."
The documentation is spread over many PDFs, incl. from different venors who use the same ISP like NXP, who tend to document things in a bit more detail.
So at worst, it's a gray box. ;)
If it was just the ISP there should have been a way in the past to take the ISP blob from similar chipset phones in the past and just use it with an updated base no? But that by itself doesn't seem to be enough to get the cam working decently.
There are simple things like debayer, but even just talking about exposure and gain control based on illumination is a pretty complex subject in itself. First you need to define parameters when to change exposure and when to leave it alone and just use camera gain. Then you need a set of parameters for when to use analog gain and when to use digital gain as communicated to the sensor. Then you have another two sets of parameters one for adjusting gain on the ISP (as opossed to all the previous stuff that feeds in to the camera chip) and some ISPs I'm told also have more advanced builtin correction based on curves. All this and we haven't even touched on how you measure the Illumination of the scene. At the very least you need one profile for back-lit and one for front-lit scenes. Then you need parameters for how the isp decides which profile to apply automatically and so on.
Mind that this is just brightness, we haven't even started talking about color correction, white balance, denoise, sharpen and 13 other ISP blocks that are there. BTW, you need another profile for every resolution your camera outputs too.
I don't know a lot about different ISPs. I only worked with one, but it gave me enough of an understanding to know why there is no way one profile will work for two camera chips. Or one profile will not work with two different versions of ISP hw.
That's why I hope libcamera succeeds. They try to standardise all the isp control, but it is a huge task especially when every single isp is different.
It's great to have more people working on imx219 though:)
Regarding my imx219 code that pushes it to 120fps I haven't submitted it upstream because the way fps is set changed in v5 Linux kernels. I used a v4 driver as a starting point because many other features only worked on v4(such as full gpu encode/decode etc). Also I preferred my userland side to not have to calculate timing parameters as it would require a per device implementation.
So my 120fps imx219 driver is for linux kernel 4.19 named rk3566_bsp_kernel-4.19, but I do have plans to upstream my code when I make them compatible with newest.
https://drewdevault.com/2022/01/18/Pine64s-weird-priorities.... https://news.ycombinator.com/item?id=32495274
Manjaro consistently applies incomplete and WIP patches into their builds, causing that sort of instability. Every time I have had a strange issue filed in a program I maintain, it has been traced back to Manjaro doing something like that.
IMO, Manjaro should take the Endeavour route and just use the upstream repositories. Manjaro does a lot right and brings some good ideas, but they are really lacking on the reliability and professionalism fronts.
Not sure what the future brings, but probably even more limitations. My next phone will probably be a "dumb phone", and all saved money I will spend on decent camera.
Highly recommended. The irony is that Googles Pixel phones are great options.
I guess the Solid Explorer's dev isn't concerned about that, because it still has that capability today.
I have never seen the whole automatic updates thing - what are you talking about there? Why would Google Play care about you making legal claims?
com.android.documentsui is included in AOSP.
com.google.android.apps.nbu.files is Google's app on the play store.
I also really appreciate seeing that for some people, there are limits to just how inferior they are willing for those experiences to be.
Very relatable. Thank you.
from my closed-source iPhone 14 Pro
That said, the iPhone 8 is still supported by Apple (probably for the last year), while the Pixel 2, like the Pixels 3, 4, and 5, are no longer supported by Google. This likely doesn't matter to someone not running standard Google-supplied AndroidOS, though!
My use of the closed-source iPhone is not solely based on feature quality, and I've got no illusions that every feature is the best in class.
P.S. Just before clicking the reply button, I decided to do the search myself, curious. The iPhone 8 was a weird model; that was the year they released both the iPhone 8 and the iPhone X, because (IMO) they lacked the courage to switch things up in their flagship product so much in one year. (Okay, and maybe also supply issues?) Still, when I DDG'd "pixel 2 iphone 8", I got results[0] that seem to not be the same as yours. The first doesn't really make any determination, the second is mealy-mouthed but seems to give an edge to Pixel 2 based primarily on price ("the Pixel 2 is a better value, starting out at about $50 and £70 cheaper than the iPhone 8" vs "you'll actually get more bang for your buck with the iPhone 8"), and the third, well, it's right in the title: "iPhone 8 vs. Pixel 2: Why Apple Beats Google." So I don't know, I bought an iPhone X that year rather than an iPhone 8, and just replaced that with an iPhone 14 Pro this year after five years of faithful service. But choosing a 2017 phone today, either one, while very clearly a step up from Pinephone for all the of the reasons mentioned in the article, maybe still reflects a but of compromise in service of wanting to run LineageOS. In 2017, was photo quality the same when running LineageOS rather than Google's own camera software? Maybe it was after that they started to apply more and more AI to the photo-taking process, I'm not sure.
0. https://www.techradar.com/news/google-pixel-2-vs-iphone-8 https://www.cnet.com/tech/mobile/iphone-8-vs-pixel-2-phone-c... https://www.tomsguide.com/us/iphone-8-vs-pixel-2,review-4779...
It does a bit, actually. While we get many updates through generic Android, some are vendor patches which stop after support stops.
At this point I think I qualify as a career Android developer (did other stuff but just a lot of time on Android) and my joking reply to anyone who asks why I use an iPhone is:
> I'll switch back when Android can handle a screen rotation without blanking the screen.
The thing is, the fact there's no smooth rotation isn't literally a blocker, but it's what such a subtle thing represents: Android apps are still living with the complexity of a situation that showed up because of the limitations of the 256 MB of RAM the Android G1 had in 2008.
-
They're still doing this messy state restoration stuff where the OS essentially has to kill part of your app to support a basic rotation, they never went back and fixed it as phones matured, bad configuration change handling easily causes hundreds of thousands of crashes a day for end users, constantly throws a wrench in development that Google has had to invent many different catcher's mitts for, drains resources that would be better used on improving the user-facing functionality of these apps.
And this isn't the only feature like that. There are so many basic sharp edges of the OS UX-wise that will just seemingly never get fixed, and even if were fixed tomorrow, it'd be at least 6 years before those improvements represented even 90% of the Android userbase (something iOS tends to achieve within 12 months...)
[and before the nitpickers show up, yes you can override configuration changes and manage that for your own app... no it doesn't scale for any "normal" app]
hm, I don't have this experience, maybe last time you checked in an old android device or something ?
Android will kill the user-facing portion of the process ("Activity") and recreate it* if you rotate the screen.
Devices have gotten fast enough to hide it with an animation, but it's still "kill UI, draw new UI". (specifically it's usually "render the UI to an image, squish it and rotate it, then show the new UI underneath".
My problem isn't with the literal animation, it's what that mess of rebuilding the entire user facing UI because they rotated their screen represents.
(* apps can opt out of this, but that's reserved for games and apps that render their own custom UI directly to a raw surface)
But this is like ground-truth stuff about Android, about as immutable as it gets: https://developer.android.com/guide/topics/resources/runtime...
The best time to fix it was before the G1 launched and the 2nd best would have been right after, but here we are more than a decade later paying for its sins.
12 years ago people were asking for this stuff: https://www.reddit.com/r/Android/comments/djp18/i_know_its_s...
This is something you learn very quickly once you start building Android applications -- or, at least, I did when I started back in 2011 (I haven't done much Android dev in the past decade since then). While I do think this behavior is a pretty terrible default behavior for the underlying UI framework, it's pretty bad that the AndFTP developers have not fixed this bug in their application -- and yes, it is a bug in their application.
The UI development paradigm has changed quite a bit over the life of Android, making "Fragments" more popular to use in some quarters than "Activities" as the "unit of UI screen" (though Fragments are still underpinned by Activities), but I believe the fundamental issue is still there: if you don't explicitly handle rotation properly, and store state in your Activity, it will get destroyed whenever the screen is rotated.
Imagine if every time you resized a window on your PC, the window got killed and restarted, and you see why bugs like that arise.
It'd be one thing if it was when the app is backgrounded, but no, even if you have one app running on your entire device, the UI be torn down.
It's insanity that in 2022 a major operating system is still doing that, and yet here we are with no end in sight.
But you know what? Maybe instead we should discourage people from clumsily trying to push their pet agenda the moment their favorite sports team (erm excuse me, Operating System) is painted in a poor light.
But we can't all get what we want can we?
Hell, I, who would be considered a power user, spend 99% of the time on my phone browsing the web, Reddit, some Facebook, using a calculator, or taking photos. Most of my computing is done on an actual computer because the form factor of a phone just doesn't satisfy my needs.
I'm just not a cool person flashing expensive iPhone taking Instagram pictures. Drop the camera suite and image processing and you can cut half the cost of a smartphone!
>Even running on the latest Manjaro build, as of September 2019, the phone is just ... crashy.
Thanks for giving me flashbacks to my Hiptop days. I loved messing with the apps, and the camera. My friends pointed out that a phone that had a flip put keyboard and apps but often crashed while being a phone was not much of a phone. :)
Luckily the Hiptop crew went on to make Android, which got to be rock solid, eventually!
https://www.androidpolice.com/google-pixel-phones-struggling...
Another article [1] from a few months back says that Rosteltelekom owns about 45% and Jolla is trying to find another EU company to buy those shares, but didn't see anything more recent.
[0] https://together.jolla.com/question/178875/rostelecom-acquir...
[1] https://www.engadget.com/jolla-samuli-simojoki-post-russia-1...
I am not here to support Stallman style political warfare "who is more open source and less working".
I am just stating the obvious. Sailfish works. Everything else I have seen is a non working joke, that is unusable. Feel free to prove me wrong.
Not with bullshit, downvoting and PRing. URL please.
Not political BSing about which project is completely open source but no one sane would use as it is completely unusable. Which is actually the topic here.
Open source is great but < working phone (less than 10 years old), with usable front and back camera, 4g, phone calls that don't reset your phone, SMS messages that are actually delivered and... oh... ah... Bruce Almighty... working fingerprint reader.
We already have a "working" Linux distribution for mobile devices: Android. But what projects like Pine and Librem are trying to do is build an alternative that is 100% open source (or as close as reasonably possible) and does not rely on proprietary, third-party services. If that's not important to you, that's fine, but don't dismiss the progress that's been made. Some of us actually care about this stuff. :)
I'm also not sure how Sailfish is relevant to this discussion at all while Android exists and is the dominant (non-iOS) mobile platform.
All this means, for a tinkerer/hobbyist one can do stuff that is simply impossible on any other hardware that has comparable performance (basically modern 64bit embedded systems like those in newest mobile phones).
Then there is all the time we(hobbyists/devs) spend tinkering with this hardware. When one invests so much time into some product it is hard to just let it go.
However, having said all that I would never use an unfinished product as my main phone. If I had a pine phone I woukd probably carry it as a secondary, but those devices are nowhere near the state people can rely on them in case of emergency.
(That aside, I have not had the problems OP describes - but the worst problems with PinePhone (outside of highly experimental software) I suffered through where with Manjaro.)
I think there is a lot of secret sauce that perhaps is lacking in the Pine software stack.
That said, the PinePhone camera app, Megapixels, recently got better post-processing thanks to postprocessd, which may have landed in OPs Manjaro or not.