Rolling Shutter Simulation in C | Hacker News Reader

Rolling Shutter Simulation in C

nullprogram.com

188 pointsingve34 comments

A couple friends of mine wrote an extremely mathematical analysis of the rolling shutter effect, and even looked at some related questions like "given a photo from a rolling shutter how can you tell how many propellor blades actually exist and how fast it is turning?".

http://danielwalsh.tumblr.com/post/54400376441/playing-detec...

>and even looked at some related questions like "given a photo from a rolling shutter how can you tell how many propellor blades actually exist and how fast it is turning?"

I've seen some code (don't remember the website now) that gets you the non-rolled shuttered (sic) image from the rolling shutter one.

(It works to reconstruct a scene, but has issues at edges etc, so it's not really usable to fix a photo/video 100% -- but there are commercial NLE plugins that do reduce rolling shutter in post).

One concrete example: the most recent version of Autopano Video Pro does rolling shutter compensation as part of its stitching of 360 video from a rig of GoPro cameras.

Thanks for posting that - nice add to my collection, and I'd argue they work out some of the math far more than others (it's implied in other places, but its nice they actually worked it out instead of another bloody exercise for the reader) :-)

The author mentioned YUV4MPEG. It's a wonderfully simple video format, here's a tiny C program to stream a video of Game of Life (With colours!):

    #define  F for(i=0;i<l*3;i++)
    main(n){int i,j,w=640,h=480,l
    =w*h,o[]={~w,-w,-w+1,-1,1,w-1
    ,w,w+1},b[l*5];F b[i]=rand();
    for(puts("YUV4MPEG2 W640 H48"
    "0 F30 C444");puts("FRAME");)
    {F{for(n=j=8;j;n-=b[i+o[--j]]
    &1);b[i+l]=(n^5&&!b[i]|n^6)-1
    ;}F putchar(b[i]=b[i+l]);}}

(You can pipe the output to mpv or a similar media player.)

Heh. I tried running that code through three different beautifiers and they all failed spectacularly.

For those of us who don't enjoy visual self-flagellation, here's an attempt at untying the above ball of yarn:

#define F for(i=0;i<l3;i++)

  main(n)
  {
    int i,j,
        w=640, h=480,
        l=w*h,
        o[] = {
          ~w, -w, -w+1,
          -1, 1,
          w-1, w, w+1
        },
        b[l*5];

    F
      b[i]=rand();

    for( puts("YUV4MPEG2 W640 H48" "0 F30 C444");
         puts("FRAME");
        )
    {
      F
      {
        for( n=j=8; j; n-=b[ i + o[--j] ]&1 );

        b[i+l] = (n^5 && !b[i] | n^6) - 1;
      }

      F
        putchar( b[i]=b[i+l] );
    }
  }

EDIT: I had no idea you could italicize code blocks! This was an accident, but I'm going to leave it because it looks pretty. Has this always been a feature? I've never seen italicized monospace before. That first line is supposed to read "for(i=0; i < l times three ..." but the asterisk is somehow causing HN to italicize the code block. There's no matching closing asterisk, so this is pretty amusing.

> Has this always been a feature?
I've seen it before so I think it's always been a bug :)

foo*bar
baz
See? No italicization.
Now I'm curious what triggers it...

foobar
baz*quux
Aha. HN's parser is matching with the asterisk in the code snippet.

It even affects the reply button/text.

A dangling asterisk -- which turns on italics mode on HN -- without the matching asterisk to end it.

I think you can fix it by putting the first line with the #define in the code block too.

That's really cool. I wanted to see what the second order effect would be (as the shutter speed changes, how does the final image change?)
Take a look: http://antidom.com/fan.webm

Nice. Reminds me of WW2 film of propeller planes. The propellors often looked bent back.

That video [1] has been motivating a lot of folks! Yesterday someone posted a Rust [2] solution to do the same thing.
[1] https://www.youtube.com/watch?v=dNVtMmLlnoE&feature=youtu.be
[2] https://www.reddit.com/r/rust/comments/6kl0vj/i_made_a_progr...

Rolling shutter might sound very bad, but has some nice side effect properties. Latency between light hitting the sensor and readout is minimized (divided by rolling shutter readout block size/row) compared to global shutter. This means your 1080p 60Hz camera can have 64KHz time resolution.
Cool research into RS effects: "Direct Semi-dense SLAM for Rolling Shutter Cameras". https://www.youtube.com/watch?v=OLD1eeu1EUI https://github.com/jaehak/rrd_slam
This only tries to reverse adverse effect of RS (at best as good as global shutter), but in theory it would be possible to exploit RS effect to boost accuracy above GS.
Afaik Jeri Ellsworth CastAR (shutdown, because Rubin invested only to sell to google, and that didnt happen) IR tracker took advantage of cheap cellphone camera rolling shutter to boost both spatial and temporal tracking resolution.

Nice! I'm curious if there's an open platform for writing video effects that would be compatible with other video editing and conversion software. Something that doesn't have any glaring inefficiencies and in C or C++.

Openframeworks - http://openframeworks.cc
It’s a c++ multimedia Swiss Army knife with a pretty strong community and repo of addons.
Libcinder is another option, though only for Mac/Win

OpenFX - http://openeffects.org

That's a really clever trick, piping the video to PPM format and then doing intermediary work using simple code. I had never thought of it. I bet you can do fun stuff like pipe it through imagemagick for fun transformations.

Can anyone explain why rolling shutter is still necessary? Wouldn't current electronics have the memory bandwidth needed to just sample the entire cmos sensor at once?

You've just hit the nail on the head yourself: bandwidth.
Sampling the entire CMOS sensor at once is not really possible when you stop to consider the sheer magnitude of bits required to represent a single still image. The CMOS sensor only has a finite number of pins connecting it to the logic board responsible for polling its state, so it makes sense to use an addressing system to read the sensor data. This addressing system uses a small number of addressing bits (each pin is usually just one bit) to move a sliding window over a much larger amount of data. Regular computer memory works the same way. This enables that data to be passed through to the camera in small pieces. While this can be made rather fast with good engineering, it is not instant, and the non-instantaneous nature produces the rolling shutter effect.
It's the physical limitation on the number of pins that creates the need for the rolling shutter in modern CMOS design. Creating speedier memory access routines can reduce the effect of the shutter, but not eliminate it.
A possible alternative approach would be to build some memory into the image sensor, so that you can ask it to "lock" all the bits in place, and then read those bits out at your logic board's leisure. That should effectively eliminate the rolling shutter effect, (More or less, assuming your "freeze" signal arrives at all the pixels more or less at the same time) at the engineering expense of needing to add per-pixel memory to your image sensor, and additional electronics to perform the lock. I'm no expert on the subject, but I would not be surprised if more modern image sensors have some feature for this purpose, especially those used in high speed cameras.

Don't confuse the timing of light acquisition with read-out at the overall focal plane level.
Numerous CMOS, CCD, and other types of focal plane / 2D detectors have global shutters where all pixels 'integrate photons or energetic particles' at the same time for a set amount of time.
Read-out is performed in a rolling fashion... the data generally has to be serialized for analog to digital conversion (although not always) and/or transmission over a communication protocol or write to file.
A global shutter detector at the hardware level has settings for framerate and light integration time (among other things). Readout time is more or less constant for a particular camera sensor. So if integration requires say 20ms, and readout takes 30 ms... you're looking at about 20 FPS image acquisition speed.
Good overview: http://www.red.com/learn/red-101/global-rolling-shutter

IIRC a typical implementation of electronic global shutter for CMOS sensors requires one extra transistor per each pixel to "lock" the charge. As for why it's not being done outside specialised sensors: not many people are concerned about this problem, they are more concerned about other parameters of image sensor.

Ooh, this is excellent! I hadn't even thought about physical shutters, that adds a whole new wrench.

There are global shutter CMOS image sensors. But it comes with a cost. After each frame is exposed the values in the photocoupled capacitors are transferred into neighboring capacitors in one go, then those are read off serially while the photocoupled capacitors are exposed for the next frame. Alternately, each pixel is locally connected to an ADC and enough memory to digitize the image all at once.
Canon, for example, announced a global shutter CMOS sensor last year. The problem here is that those extra components come at a cost. Either they crowd out existing components on the sensor, which would dramatically reduce light sensitivity. Or, they require additional layers in the CMOS die, which significantly increases fabrication cost due to design costs, fabrication time, lower yield, etc. Additionally, the extra step in digitization reduces effective frame rate.
[1] http://global.canon/en/news/2016/20160831.html

Yes, attaching memory to the CCD, with a latch clock to function as a shutter, would prevent the rolling shutter effect, bandwidth be damned. But that too is an extra expense, so you're not likely to see it on low end cameras (e.g., phone cameras).

>But that too is an extra expense, so you're not likely to see it on low end cameras (e.g., phone cameras).
Not so sure. Phone cameras are big selling points for phones, and have some hardware innovations (and access to much faster processors than the average "high end" DSLR/mirrorless), that , if we ignore the smaller by necessity optics, are in the same or even better league tech-wise to the average Sony/Canon/etc. Talking of course for flagship iOS/Android phones, not the average phone.

Cell phone cameras can produce 12-megapixel images, and the data from the sensor is uncompressed. It's very large. Sampling all pixels at once would require much larger internal memories in the image processors, which would make the chips more expensive and draw more power.
Only a few use cases would benefit, so the there's no economic incentive to change how it works.

you dont need to ADC every single pixel at the same time, you sample and hold current values (buffer) and ADC in the background while the new image is being captured.

Recent Sony Alpha 9 camera has quite a revolutionary mirrorless system with global shutter on a 24MPix full frame sensor capable of 20 RAW frames per second with up to 1/32,000 sec. shutter (not affiliated with Sony, just impressed by the camera). So it is already possible, but a bit pricey at the moment.
Edit: technical details based on dpreview claims it's actually not a global shutter, just a very quick rolling one. "...this capability stems from a stacked CMOS image sensor, which includes processing circuitry nearer the pixels and features built-in memory to deliver all this data to the off-board processors at a rate they can cope with. It's this structure that enables the camera to shoot at 20 frames per second and do so with an electronic shutter that's fast enough to minimize the rolling shutter effect."

>Wouldn't current electronics have the memory bandwidth needed to just sample the entire cmos sensor at once?
Don't CMOS sensors increase in pixel density (from a few megapixels to 30 and even 50 mp in some Sony full frame cameras today)?
If the camera sensors were made as just 4K resolution sensors it might be easier to scan it all at once in fast enough time (though we also increasingly ask for more slow-motion capabilities, which requires even faster frame rates).

One issue is dynamic range - rolling shutters tend to have a higher DR

This is great. Searching for an example of the rolling shutters PPM image transformation in Python, anyone?

Reply on news.ycombinator.com