Gcam, the computational photography project that powers the Google Pixel camera | Hacker News Reader

Gcam, the computational photography project that powers the Google Pixel camera

blog.x.company

196 pointsPleaseHelpMe75 comments

> So the team started to ask — what if we looked at this problem in an entirely new way? What if, instead of trying to solve it with better hardware, we could do it with smart software choices instead?

I feel like the lofty language takes something away from the article. It's not like using software to improve image quality is anything new. The image stacking technique they describe isn't particularly novel. I've done stuff like that for years. It even works on 3d renderings from a path tracer. Its cool their camera is fast and provides a nice ux when stacking images, but that's only possible because of the cameras hardware.

Agreed. I think my last 5 smartphones have all had an HDR mode. Claiming it is something new is a little strange.

Even HTC's relatively new technique of recording a couple seconds of continuous video as long as the camera app is open so it can provide an "instant" shutter button is years old at this point and that came out long after HDR in smartphone apps.

What's the resolution of the sensor? Does that mean it can record video at 16MP or something like that? I find that surprising given that all handset makers make a big deal of 4k recording.

Guess: video recording produces large amounts of data while recording, so is disk write bound. "Instant capture" requires only a short, fixed duration, recording, so is only memory write bound.

An alternative possibility, along the same lines, is that recording durable video is CPU bound due to compression, but a temp cache in RAM is small enough to have minimal compression requirements, so is not CPU bound.

Camera HDR modes usually take 10 to 30 frames at full sensor resolution (RAW).

That comes out to 16 Mpixels * 12 bits/pixel * 30 = 720Mbytes of RAM. Doable just about for a smartphone.

If RAM is a concern, by using custom silicon which can process HDR very quickly, you can start to discard the first frames before the last is taken. The hardware necessary shares a lot of logic with video compression (calculating motion vectors etc.)

It's not the same as "normal" HDR:

https://research.googleblog.com/2014/10/hdr-low-light-and-hi...

I read the article. That is absolutely normal HDR. I was doing most of those techniques five years ago via DSLR burst and merging the images in post.

What from that article don't you consider "normal HDR?" Because as I said, everything listed there is completely par for the course. "HDR+" is just obvious branding for standard HDR that happens to use more bracketed shots than other HDR systems (e.g. 5x instead of 3x).

Normal HDR is usually implemented as bracketing. As the section "Capturing high dynamic range scenes" explains, HDR+ does not use bracketing.

That article describes exactly how "normal HDR" is achieved with a DSLR.

I don't think "normal HDR" traditionally includes this:

"HDR+ addresses [low-light blur/noise] by taking a burst of shots with short exposure times, aligning them algorithmically, and replacing each pixel with the average color at that position across all the shots. Averaging multiple shots reduces noise, and using short exposures reduces blur."

Reducing shutter speed to reduce blur isn't new (https://www.dpreview.com/articles/1528656007/notimagestabili...), accumulating sensor exposure is how DSLRs work, and aligning frames is how image stabilization works. It might not be what we call "HDR", but there's nothing new here. The combination in a phone camera sounds nice.

Normal HDR combines regular, underexposed and overexposed shots. HDR+ does not.

Yes but a lot of cameras have low-light mode which takes multiple shots (not bracketed).

I'm still not seeing how this isn't a fairly simple combination of existing techniques.

That's true. It's an implementation of several similarly old techniques. Imaging and image processing scientists have been doing this stuff for decades.

From the results I think they did a great job, but at least for the parts I've heard discussed in any technically detail they haven't really broken any ground in the imaging sense - what they have done is apply it to consumer electronics. They aren't the only ones doing this, mind, but they've done a good job on a mass produced platform. So that benefits a ton of people.

Really it is sort of an "obvious" next step, because physics is limiting what the sensor & optics can do while packaged in the size we seem to have converged to on a phone.

Especially considering that Nokia gave a huge push to computational photography back with the N900. In fact, the name "GCam" is probably related to "FCam".

I guess this is just how marketing copy ends up being handled at companies this size. Apple constantly insinuates that they wrote WebKit from scratch, and that it constituted some sort of "innovation".

I'm not even sure what that means. So Google would rather use the same sensor hardware for 4 years in its Pixels, and just focus on the software? I'd rather they renew the hardware with something better as well, as the technology improves.

Sigh. I'm disappointed that this piece seemingly gives no credit to Andrew Adams, Jiawen (Kevin) Chen, and all the folks that made gcam happen. There isn't even a mention of Halide...

Edit: just to be clear, I like Marc just fine, but I don't see the point in excluding the team.

Disclosure: I work at Google and know the folks involved.

It's a desperate attempt to make X look good. There haven't been too many successful "graduations".

What rate of success would you consider acceptable? I thought the whole point of Google X was basically "let's throw stuff at the wall to see what sticks", so it seems pretty much expected that not a lot of products would make it out of the lab before being killed.

Look through most MegaCorp press pieces, and you'll find that the individual team members rarely get shoutouts. It is disappointing.

Surprised by Negative Tone of Comments.

I have been using PixelXL since launch and every one of my friends who looked at photos are amazed by the camera (Many of iPhone Users). PixelXL made me default photographer at parties (And fact that I can share Photos using Google Photos with everyone at Party).

Pixel Camera is amazing compared to any existing SmartPhone camera.

Side Note: The only issue I have with Pixel is that its easy to break glass.

They aren't negative about the Pixel but rather the idea that it was only software wizardry that was the cause of the camera quality improvement.

Not difficult to beat the iPhone camera.

Why does my iPhone 4S take better photos than my iPhone 7? Modern iPhones photos are smeary smudgy mess. Can always tell on Reddit or Facebook when an image has been taken with an iPhone.

Luckily using RAW mode helps, which means it's some lunatic software noise-reduction in newer versions of iOS that is the problem.

I honestly think that you might have a lemon. It has been my experience that iPhone 7 images are among the best in class (I can't comment on the quality of the Pixel's camera, but I've heard good things). I understand your grievance, but honestly it's impossible to posit that an iPhone 4 takes better pictures than an iPhone 7.

My iPhone 4 takes unbelievable macro pictures. I hate to say this, but better than my Canon 20D, with an L lens. I was thinking about buying a macro lens for Canon, but can't justify the it anymore.

Just macro in my world though. The dslr still takes better portraits, and nature shots.

Maybe when compared with iPhone or Nexus cameras.

> Surprised by Negative Tone of Comments

Welcome to Hackernews :)

Hackernews, where everything sucks and do you have numbers to back that up ?

I've had the same reaction, but with an HTC 10. I'm curious now what a side-by-side between the two would look. The 10 uses a really novel lens with OIS and HDR, whereas the Pixel has no OIS but uses fancy EIS.

I'm actually surprised how little love the 10 gets, considering the noise made around the hardware quality of the Pixel, and the fact that HTC made both. I guess people just like the Google brand or something.

I have the HTC 10 and it's easily the best phone I've ever used. The only thing I didn't like about it was the amount of software you need to disable before it's usable. Like all those HTC speciality apps you can't uninstall and who want to send your tracking data. It's not as bad as some other manufacturers, but definitely annoying after using OnePlus products.

Seems like the HTC 10 really isn't all that great.

"The real surprise from these tests isn’t the winners, though — it’s how bad the losers are. In some situations, the LG, HTC and Sony phones put out pictures that loom downright unusable, something I’d expect more from a 2013 Moto G than "

http://bgr.com/2016/12/14/iphone-7-camera-vs-google-pixel-ga...

Well that's just a reference to http://www.techradar.com/news/how-2016s-best-camera-phones-c...

It looks like they had smudges on the camera lens. I've taken much more challenging shots than that without the horrible blur they have. But smudge any smartphone lens with fingerprints and it will look like that. Hell, smudge a DSLR lens and it will look like that too.

The technology is fine (although nothing novel is shared in this article), the negativity is directed at the extreme big-noting tone of this marketing message for X.

What about photos taken by the Windows Lumia phones by Nokia? I had a 920, when it was brand new I thought it took great pictures. It wasn't the 41MP camera and even that something about 'not real pixels' or multiplied pixels, can't remember.

Your anecdotal experience is not especially relevant to the conversation. It's great your friends think your phone is cool. Literally every person you ever meet in your entire life could tell you the pixel is awesome and that's still too small a sample size to be indicative of anything at all.

This blog post is remarkably light on details. For the actual research paper behind the software in the Pixel camera app: http://www.hdrplusdata.org/hdrplus.pdf

Surely many of these improvements are simply due to the Sony Exmor sensor? HDR processing is only really possible when you have fast (and full resolution) burst mode available on the sensor.

Given the relatively long timeframe of the project, we don't know if that was an idea/requirement from the gcam side.

> One direction that we’re pushing is machine learning

Is machine learning the answer to everything these days? Is it over-hyped...or should I start worrying that I don't know anything about it (as a developer I mean).

I don't think it is the answer to everything, but it can be very broadly applied to a whole lot of scenarios and get pretty good results for the most part. I don't think you should worry, however you should take a look at it because it is pretty cool!

A lot of the modern algorithms we write have to do with pattern matching (eg patterns in numerical data, pixels, etc). Machine learning is basically a meta technique for pattern matching. It should probably be in your toolbelt, even if it's not your main field of expertise.

The limits of ML (as we know it) is still not fully known, so it's worth throwing a wide range of problems at it to see what sticks. I think that's why you see talk of ML everywhere, because people are still feeling out the space, seeing how far ML can go. Eventually, the dust will settle and the next big thing will come along.

So yes, should start worrying?

ML is delivering a lot of value, but is largely being applied to high value areas since the start-up costs of adding ML to anything are pretty high compared to traditional dev work. So it's not going to replace traditional dev work any time soon, but it's certainly going to be important.

Had an interesting/cool moment with my Pixel at the theatre today. There was a line to get your picture taken in front of the show's giant poster. The attendant would take your phone and snap a pic.

When the attendant used my phone, she couldn't tell it took a picture -- because when you press the button it's instant. She was so used to other phones taking a second or two before a picture was taken.

Just tried on my iPhone 7, it's definitely instant. Perhaps that's more of a UI feedback flaw in the Pixel then?

Article does not deliver on the headline (of course). It says that X wrote software that became HDR+ , Google's version of HDR.

Please > Most recently, Gcam’s HDR+ technology launched as the default mode for the critically acclaimed Google Pixel phone. DxOMark, the industry standard for camera ratings, declared that the Pixel camera was “the best smartphone camera ever made” in 2016. Reflecting on the evolution of the project, Marc says, “It took five years to get it really right…and we’re grateful that X gave our team the long-term horizons and independence to make that happen.”

There's a lot of pressure on smartphone makers to make really good image processing software because they are extremely restricted hardware wise -- there's just no space for a proper lens and a big chip. And it's amazing how far they've gotten in the past few years. Phones are now better cameras than most point-and-shoots.

I wonder if there's any chance that some of that amazing software makes it's ways to pro cameras? It seems that camera companies can't keep up.

For example, the autofocus and whitebalance on my phone is almost always spot-on and perfect. My 'real' camera (a Fujifilm X100T) often gets the focus wrong, and the automatic white-balance is not as reliable. It seems like camera makers just can't keep up on the software side.

[Heh, the cynic in me is amused when these things come out on the heel of a lot of bad news about Google.]

I think computational photography is perhaps the biggest change to picture taking since Ektachrome. Seriously it takes pictures with existing hardware that are better pictures and does it by applying some interesting science to the mechanisms in the pipeline. I've been very impressed with the results and how rapidly the camera comes up with the image after taking it. I wish my Canon SLR had this as part of its software load.

What's the 'lot of bad news'? I couldn't find anything on HN search recently, except discontinuation of Google talk. Perhaps you confused Google with Uber?

Several articles covering advertisers pulling ads from Youtube[1,2,3] because Google runs them next to objectionable content, a 4% drop in Google's share price [4,5]

[1] http://www.theverge.com/2017/3/22/15029214/att-verizon-googl...

[2] https://www.theguardian.com/media/2017/mar/17/google-pledges...

[3] https://arstechnica.com/business/2017/03/big-us-companies-pu...

[4] http://www.cnbc.com/2017/03/20/google-stock-downgraded-to-ho...

[5] https://www.recode.net/2017/1/26/14402080/alphabet-google-ea...

Marc Levoy (who the article is about) gave a lecture series [1] at Google about digital photography.

[1] https://www.youtube.com/playlist?list=PL7ddpXYvFXspUN0N-gObF...

I like how they have taken credit for implementing HDR along with a smartphone-sized burst camera.

This is purely a technical implementation and is not novel in almost any way.

I agree that this article was thoroughly disappointing. I went in expecting a lot and received VERY little information.

Lots of PR-ish tone here. Image fusion is pretty standard in video cameras, it's part of the imaging pipeline. The IP on that dates back 15-20 years.

It's a step in the right direction for mobile photography but not really revolutionary. Exposure fusion and pixel alignment has been around for a long time.

Would like to see more options for tonemapping (really like ACR's highlight/shadow recovery), higher dynamic range sensors and most of all 10bit or raw video on mobile phones.

Halide is great btw

Ok, since maybe Google people will be looking at this:

I have a Nexus 5x. The Google camera app recently, finally, received a revision that fixed some significant bugs, e.g. crashing on "zoom out".

It also introduced ISO offset/adjustment. However, the widget to make that adjustment appears only briefly, at the top of the screen, immediately after touching the screen -- to set a metering point or to zoom. This makes that control both difficult to trigger, at the top of the screen, and difficult to use with its small scale and position that leaves one's finger obscuring the readout. On my 5x with a Pleson glass screen protector, the resulting value also tends to jump a bit as my finger is removed (1). Finally, the setting only remains for a matter of seconds, until the app clears any custom (i.e. via touch point) metering setting (2).

I like to adjust the ISO offset setting until I get the exposure I'm after on the screen. I wish the control was easier and always available to trigger, easier to use, and that the value set would persist at least until the shot is taken and preferably, or optionally, until it is manually changed back.

You finally gave me ISO offset control -- thank you! But the control is difficult to the point where I mostly don't even try to use it.

The app update also introduced a widget along the right side that displays an ungradated meter of (digital) zoom level. Beyond seeing the relative zoom setting, this control doesn't really do much that I find useful. (Actually, sometimes I find myself waiting for it to clear so that I can see my composition better.)

That widget position on the right side would be the perfect place to optionally display a touchable ISO offset scale. More real estate, making fine adjustments easier. Finger not in the way of seeing both the values and the shot, while adjusting it.

If not by default, then maybe something that could be selected in the options?

Friends really like my photos from the Nexus 5x. I'd love to have more-better controls when composing them. Thanks.

1. My zoom level can similarly jump a bit when I lift my fingers from the screen. I assume this is because of the screen protector, as opposed to the underlying phone/app, and have come to live with it as the price for the screen protection. The jump in ISO offset, maybe because of the small size of its widget as well as its placement, is harder to control.

2. This momentary clearing was already happening before the app update. I like to take my time composing my shot, and this clearing already made doing so somewhat frustrating at times.

So it's just the same HDR+ they've been using in Nexus devices for a long time now? They didn't do anything special with the Pixel's camera, specifically?

Sorry to hijack the thread but I found your handle a bit disturbing.

What do you need help with?

He created his account 231 days ago. Whatever he needed help with presumably is no longer an issue.

It never hurts to ask. Maybe he's been waiting for this day for 231 days?

Picking a handle.

Haha it's just a name I usually use.

The surprising part about this is that X decided to go with Medium as their publishing platform.

Let me continue: I specifically dislike how medium breaks urls.

Do you mean the .#<alphanumeric-code> at the end?

I'm surprised Google wouldn't also use their own URL-breaking property, AMP. :)

Or their other url-breaking platform - blogger - who found out it was a good idea to redirect to a different top level domain based on visitor location :-/

How does it break them?

As far as I can see, the only bit that matters is what appears to be a hash at the end. In this case: 3ee86657d6c9. That is, https://blog.x.company/3ee86657d6c9 is the same post.

I don't know how this is substantially+notably different to how other blog platforms do urls.

Problem is everone gets a new url. Haven't foun a good way to check if one link to medium is the same as another.

This breaks a number of things like duplicate checking for news aggregators and tag suggestions for bookmarking services.

Edit: checking now it seems at keast the part before hash stayed unchanged after copy-pasting into another browser and loading. Have they changed or have I misunderstood the whole time?

I don't think they've changed; if they have, it wasn't recent. Stripping the hash doesn't break anything that I've noticed - I assume it hoses their "share this story" tracking, but I don't really care about that.

WHO WROTE THIS?

CATHOLIC

We've banned this account for breaking the HN guidelines. If you decide you don't want to be banned, you're welcome to email hn@ycombinator.com and promise to follow the rules in the future.

Reply on news.ycombinator.com