Comment by Roark66 | Hacker News Reader

Comment by Roark66 | Hacker News Reader

The reason why modern smartphones have such good camera picture quality is primarily thanks to the ISP profiles developed by teams of people for the particular sensor in use. ISP is an imaging signal processor, a component of most mobile SOCs that sits between the camera and the rest of the system. It is a proprietary component that is essentially a black box controlled with a "profile" for specific camera chip. The profile specifies how to debayer, denoise, sharpen, color correct, adjust brightness and autofocus of the chip. Without a very good profile even a good camera sensor output will look like crap.

I've spent few weeks of my (spare) time improving a Sony imx219 driver for a Rockchip rk3566 cpu that is used in few pine devices I own. I needed high fps video (I managed to get 120fps 720p - I think I could push it further to 140 perhaps, but this was OK for now) and as low latency as I could get. I wanted <20ms as I could encode 720p h264 frames in 7ms,unfortunately I got stuck with ISP (image signal processor) adding 40ms of latency even when in pass through mode.

The modern ISP is a "black box" even with source code for the drivers. In addition every single ISP has its own proprietary format for those profiles. There are OS projects that try to make universal camera sensor profiles that work with multiple ISPs (libcamera for one), but it is still early days.

What we need is a truly programmable/flexible isp. As it stands the only way to get sub 40ms of latency from a camera for a hobbyist is to use an fpga...

People used to make similar arguments about injket printer drivers (where the computer does dithering, head scheduling, etc).

Over time the open source crowd implemented the state of the art image processing techniques once, then pointed them at reverse engineered backend drivers for each model.

The result was better than what the manufacturers (especially HP, but also canon, etc) could do.

I'm hoping history repeats itself with these camera sensors.

The Pinephone Pro ISP is documented in RK3399 TRM (part3), and in various PDFs available online. It's not really a huge blackbox.

It does not make the calibration (or preparation of profiles for various light conditions) easy though. There's a need for a bunch of HW tooling that is somewhat expensive for a hobbyist. And there's a lot of work and algorithm design that needs to be done outside of ISP for controling the autofocus, autoexpsoure and detection of type lighting in the scene and profile switching. None of this is particularly trivial.

It's very clear he is talking about black boxes from the perspective of one trying to reverse one to fix pinephones isp...

Respectfully, the person you are responding to knows what he is talking about: https://xnux.eu/log/

So they write a bunch of drivers and stuff for pine phone. As a pine fan I thank them, but this doesn't mean we can't have a difference in opinion what does it mean for some component to be a black box.

If you compare with some other manufacturer that gives us absolutely no docs, then yes having register level docs and user side source is really nice. But if you want to do something unconventional (like switch the whole damn thing off and bypass its latency) not knowing how it works internally is still a black box. For example there is an irq that fires when the first component of the isp(MI-memory interface) sees a vsync signal for a frame. We know it exists. We know how to enable/disable that irq. We can write code that does stuff when it fires. But what we don't know is is this vsync signal fired truly at the input of the ISP? It's output? Or perhaps between some specific components? It turns out that vsync fires not when the isp sees the very first vsync what the location of the registers suggests, but after all the processing is done at the output buffer (40ms later usually), but one wouldn't be able to tell without a lot of profiling. This amongst other things is what I call a black box.

However, compare it with let's say a raspberry pi, and at register level the documentation is much better (for the isp).

https://megous.com/dl/tmp/3add845b7e814de9.png

There are bunch of interrupt statuses around frame capture: RIS_FRAME_IN, RIS_FRAME, RIS_V_START.

Not sure why you'd decide to use V_START for deciding when the frame is in memory. VSYNC start is sent at the start of a frame and it's documented as just a interrupt that is directly triggered from the MIPI CSI sync signal in the datasheet.

quote: "The incoming hsync and vsync control signals from the camera are connected to interrupt logic. It is possible to trigger on these signals for an event triggered configuration during processing."

Parts of the ISP are blackboxes (debayer), other parts are quite trivial and documented in a bit more detail (gamma/degamma, BLC, color matrix transformation, etc.) and some parts are a bit of a mix.

The documentation is spread over many PDFs, incl. from different venors who use the same ISP like NXP, who tend to document things in a bit more detail.

So at worst, it's a gray box. ;)

It's registers are documented yes, but from the point of how it actually works. How data flows, how it is buffered internally, how it is timed etc it is absolute mystery. Yes, you can configure various parameters, but you can't for example skip functional blocks and get data quicker. It is unknown if it is all done with hardcoded static logic blocks, some sort of purpose made vector cpu and firmware. No one knows(or those that do don't say).

I worked with rk3566, but I believe rk3399 has the earlier version of the exact same isp. The version in rk3566 is 1.1. There are also versions 1.0 and just 1. All are documented in TRMs in the same way.

Edit: BTW, if you need the tooling (Rockchips isp profile development tool v2.1 and the manual for it) please let me know and I'll send it to you.

> Edit: BTW, if you need the tooling (Rockchips isp profile development tool v2.1 and the manual for it) please let me know and I'll send it to you.

It's not usable for RK3399 version of the ISP. BTW the same/very similar ISP (to RK3399) is used in NXP SoC, and it has documentation there, too. (NXP docs tend to be better ;))

Can someone dive more deeply into this? I'm asking because it seems to me that there is more to this than what you just described. Sony AOSP is constantly upgrading the kernel, but the camera is always mediocre at best.

If it was just the ISP there should have been a way in the past to take the ISP blob from similar chipset phones in the past and just use it with an updated base no? But that by itself doesn't seem to be enough to get the cam working decently.

From the pictures linked in the article it seems to be about more than just the ISP.

People don't realise how much stuff the ISP does. It took me over a week to make an extremely basic profile for a new camera chip. And I had all the tools and the documents OEMs use. There is a 100 page long procedure I had to follow taking multiple pictures of calibration standards illuminated with various calibrated lights. Then at the end it still looks like crap, but it is pretty true looking. Then the work to make it look good really starts. I had no idea before I started.

There are simple things like debayer, but even just talking about exposure and gain control based on illumination is a pretty complex subject in itself. First you need to define parameters when to change exposure and when to leave it alone and just use camera gain. Then you need a set of parameters for when to use analog gain and when to use digital gain as communicated to the sensor. Then you have another two sets of parameters one for adjusting gain on the ISP (as opossed to all the previous stuff that feeds in to the camera chip) and some ISPs I'm told also have more advanced builtin correction based on curves. All this and we haven't even touched on how you measure the Illumination of the scene. At the very least you need one profile for back-lit and one for front-lit scenes. Then you need parameters for how the isp decides which profile to apply automatically and so on.

Mind that this is just brightness, we haven't even started talking about color correction, white balance, denoise, sharpen and 13 other ISP blocks that are there. BTW, you need another profile for every resolution your camera outputs too.

I don't know a lot about different ISPs. I only worked with one, but it gave me enough of an understanding to know why there is no way one profile will work for two camera chips. Or one profile will not work with two different versions of ISP hw.

That's why I hope libcamera succeeds. They try to standardise all the isp control, but it is a huge task especially when every single isp is different.

A pine rk3566 device with a camera?

It's great to have more people working on imx219 though:)

Well, with a camera port. Both the soquartz and Quartz64-A have csi connector for the camera. The pinout is such that raspberry pi v2 camera (imx219) can be connected directly to soquartz. However for Quartz64-A it requires an adapter.

Regarding my imx219 code that pushes it to 120fps I haven't submitted it upstream because the way fps is set changed in v5 Linux kernels. I used a v4 driver as a starting point because many other features only worked on v4(such as full gpu encode/decode etc). Also I preferred my userland side to not have to calculate timing parameters as it would require a per device implementation.

So my 120fps imx219 driver is for linux kernel 4.19 named rk3566_bsp_kernel-4.19, but I do have plans to upstream my code when I make them compatible with newest.

Reply on news.ycombinator.com