Night Sight for Pixel phones
theverge.com
theverge.com
What's really cool is you can see him talk about a lot of these ideas well before they made it into the Pixel phone
Plus, if you're at all curious about the technical details for how exactly something like Night Sight is implemented on the Pixel, understanding what Fourier transformations are and how they are utilized is vital.
https://ai.googleblog.com/2017/04/experimental-nighttime-pho...
And by the original researcher in 2016:
Alan Kay https://news.ycombinator.com/item?id=11939851 Michael Siebel https://news.ycombinator.com/item?id=13895362 Ny AG Eric Schneiderman https://news.ycombinator.com/item?id=15853374 Scott Aaronson https://news.ycombinator.com/item?id=17425377 Sam Altman (he's done a few) https://news.ycombinator.com/item?id=12593689
Is there anything expected to be released in the next few months that will be in a similar price, feature set and weight class?
That is absolutely impressive.
The color and text on the fire extinguishers along with the texture detail seen in the headphones in the last picture are just stunning. Congratulations to anyone who worked on this project!
Object detection alone won't give you sharp text in low light. You need a minimum number of photons hitting pixels.
It's too bad that the technology is proprietary. I'm curious what could be done with a larger-sensor camera, from compact cameras to DSLRs.
I'm guessing that it works similarly to low-budget astrophotography but with the computer doing all the busywork for you: when you want to photograph stars or planets and you don't have a fancy tracking mount to compensate for earth's rotation you'll have very mediocre results with long exposure. If you expose a lot to see the object clearly then you get motion blur. If you use a shorter exposition to reduce the blur you don't have enough light to get a clear picture.
One solution is to take a bunch of low-exposure pictures in a row and then add them together (as in, sum the value of the non-gamma-corrected pixels) in post while taking care of moving or rotating each picture to line everything up. This way you simulate a long-exposure while at the same correcting for the displacement.
An other advantage is that you can effectively do "HDR": suppose that you're taking a panorama with the milky way in the sky and some city underneath it, with a long exposure the lights of the city would saturate completely. With shorter exposures you can correct that in post by scaling the intensity of the lights in the city as you add pixels (or summing fewer pictures for these areas). This way you can effectively have several levels of exposures in the same shot and you can tweak all that in post. In the case of the city/milky way example you'll also need to compensate for the motion in the sky but obviously not on land which is also something you can't really do "live".
I have a strong suspicion that it's basically what this software is doing: take a bunch of pictures, do edge/object detection to realign everything (probably also using the phone's IMU data), fit the result on some sort of gamma curve to figure out the correct exposition then add color correction based on a model of the sensor's performance under low light (since I'm sure by default under these conditions the sensor will start breaking down and favor some colors over others). Then maybe go through a subtle edge-enhancing filter to sharpen things a bit more and remove any leftover blurriness.
If I'm right then it's definitely a lot of very clever software but it's not like it's really "making up" anything.
But do we know that it runs on phone hardware? If voice interfaces have taught us anything, it's that we can't ever make that assumption again.
The amount of data you'd have to send to run this off board would be enormous, but hey, anything for a jawdropping hype feature, right? It just works, those preview pictures literally made me check the price and size of the pixel 3, and I haven't been interested in anything but a Sony Compact since the z1c came out.
Chemical reactions by bacteria breaking down food produce light, enough for humans to see in only the darkest of places (if you live in a city, you won't ever encounter dark enough situations).
A camera simulating a 1 hour exposure time in a closed refrigerator ought to be able to see it pretty easily.
For meat we've got freezers. Is ready food going spoiled a big issue?
Not sure how it would increase bacteria, as bacteria don't photosynthesis light, unless the lights gave off sufficient heat to increase their activity. If anything I would think the light might inhibit bacterial growth.
Turn one LED on, while the other stays off.
Observe the leaves of lettuce after a week.
But I didn't find anything on bioluminescence occurring naturally in the kinds of bacteria you'd want to be warned about. Did you ever personally see glowing food?
[1] http://cdn.intechopen.com/pdfs/27440/InTech-Use_of_atp_biolu...
https://en.wikipedia.org/wiki/Luciferase
"Luciferase is a generic term for the class of oxidative enzymes that produce bioluminescence"
and,
"Bioluminescence is the production and emission of light by a living organism"
It's very faint and would be difficult to notice without trees to shield it from moonlight. A camera could pick it up with a long exposure.
I don't know if its really that bad of an idea, but we didn't allow projects that had been done before so they were all rejected.
(And in the end, that's better: an inventory-tracker app that's on your phone is able to tell you to throw stuff out without you needing to own a "smart-home hub" or configure your fridge to connect to your wi-fi; and, unlike the fridge, its notifications will probably keep working even if its manufacturer goes out of business.)
Some city folk have doors and window shades. My old apartment kitchen was on the windowless side of the apartment with a door. If you close the door (and unplug the microwave), it was pitch black. Though I never saw any glowing food, not even the spoiling fruit on the counter.
(I'm talking about developing sheet film in trays, btw... you don't need a darkroom to develop rolls or 4x5 sheets.)
I'd be interested to see how night mode performs when objects in the frame are moving (it should work fine, since it will track the object), or changing (for example, turning pages of a book - I wouldn't expect it to work in that case).
I must imagine that the sensor is doing an extra but un-perceptible long exposure than then is used to correct the lightning of the dark version.
That said, the effect of some of these photographs is striking, and I'm sure the tech is interesting.
See: https://www.celsoazevedo.com/files/android/google-camera/
Upgrading from a 3-year old Samsung S6, where I could almost see the battery percentages drop off percent by percent, the P20 Pro's 4000 mAh battery has been great (too bad the wireless charging didn't appear until the new Mate P20 Pro).
Except the Huawei does and in actual same-setting situations the results are better than the Pixel.
Did you actually watch the video?
The P20 Pro does exactly what this new night mode does, albeit did it months ago. It does image stacking (which is not a new approach). In direct comparison testing -- in that video -- it yields better results.
This whole discussion has just been an bizarre.
You are completely and utterly wrong. Yet you continue. Amazing.
And you're not the first to distract with claims that it is someone else's ignorance.
The P20 Pro does image stacking. Period. That is exactly what this new Pixel mode does. In actual results the P20 Pro is better.
I'm not the person you've been talking to, but I don't think I'd agree with that statement. To take the video you linked earlier for example, the Pixel frequently gives better results. For example, this one shot https://youtu.be/wBKHnKkNSyw?t=227
Note that the Pixel 2 has a much smaller sensor, and the exposure time on the P20 is 18 times longer, and yet the Pixel generates a much better sharper image. What you're saying is correct, that the P20 is using some very advanced image stabilization to get results that good from 6 seconds of data, but the Pixel seems to clearly offer more advanced software.
Further, you're reading entirely too much into the exposure times. They are artificial and either camera software can choose to put whatever number they want in there. The aggregate time. The average time. The theoretical equivalent time. Etc. It is not the actual times.
I took actual photos with the pixel 2, and I know the actual time taken. It's less than 3 seconds every time. By all accounts and reviews I've seen, the P20 is 10-25 seconds. Show me a review saying otherwise.
This thread is dominated by Pixel...fanboys and Googlers. You can be as wrong as you like, it doesn't really bother me. But your bull headedness about your absolute wrongness is simply spectacular.
Is a pure software solution even reliable enough under these conditions? Slowness can be worked around by doing it in the background, and you get a notification when it's complete. Some people would be okay if that's the only way to get photos they wouldn't otherwise be able to get, short of buying an SLR.
I really want to know how that works for people! 99% of photos I take are of people, and the lighting is always bad.
Are there any photos of people?
I wonder if this technology will eventually supercede military night vision goggles. Having the ability to add color perception at long distances could have useful for identifying things at night.
"Google’s Night Sight for Pixel phones will amaze you"
Pre-OIS Google did this with image stacking which was a ghetto version of a long exposure (stacking many short exposure photos, correcting the offsets via the gyro, was necessary to compensate for inevitable camera shake). There is nothing new or novel about image stacking or long exposures.
What are they doing here? Most likely it's simply enabling OIS and enabling longer exposures than normal (note the smooth motion blur of moving objects, which is nothing more than a long exposure), and then doing noise removal. There are zero camera makers who are flipping their desks over this. It is usually a "pro" hidden feature because in the real world subjects move during long exposure and shooters are just unhappy with the result.
The contrived hype around the Pixel's "computational photography" (which seems more incredible in theory than in the actual world) has reached an absurd level, and the astroturfing is just absurd.
I honestly don't know which part you're doubting. Long exposures? Do you doubt that other cameras can do long exposures? Do you doubt that they can do noise reduction? Do you doubt that OIS allows for hand-held long exposures, especially on wide-angle lenses? What are you doubting, because these are all trivial things that you can validate yourself.
As to examples, you're wide-eyed taking a puff piece with some absolutely banal examples and exaggerated descriptions -- and zero comparable photos from other devices -- by someone who apparently knows very little about photography. How should I counter that? I can find millions of night streetscape photos that absolutely blow away the examples given.
Generally if you're going to pander to a manufacturer, you at least talk about things like lux. In this case it's just "look, between this setting and that setting it's different, therefore no one else can do it".
The Huawei P20 has a night mode that in many cases is superior. So much for the "period". Further it's gimmicky and has downsides that make it pertinent for a tiny percentage of pictures. Which is why it's hidden behind a "more" option. Apple doesn't have it because Apple is about making everything easy for the 99%.
Image stacking is not difficult. It is not new. Image stacking is effectively a long exposure (I've said this probably a dozen times, but still it's like people are correcting me), with some unique advantages, and some unique disadvantages. In every one of the examples given it is indistinguishable from a long exposure.
https://www.reddit.com/r/GooglePixel/comments/9qzyry/pixel_2...
I know this is true because I've tried it. I have the camera loaded on a pixel 2, and the iPhone is pitch dark in images where the pixel is fully illuminated with night sight.
More details here: https://www.anandtech.com/show/12676/the-huawei-p20-p20-pro-...
And remember anandtech did not have night sight. The p20 quality is attributed to the larger sensor.
"The Huawei does not have anything close"
The P20 Pro's night mode is arguably better than the Pixels. It is sharper, works in worse light, and has more natural colours. You also seem to be confused into thinking that DXOMark enables every special mode. They don't. The reviews are overwhelmingly simply the auto settings.
The argument I'm seeing in favor of the Pixel generally is "waves hand {magic AI!}". Sorry, I don't care how much HN is infected by Googlers and Pixel fanboys, there is zero evidence of any magical AI in the Pixel results, and they look absolutely bog standard.
Gallus looks interesting, I haven't heard about it before. Any thoughts on when you're bringing it back?
Why? Because 99.9999% of smartphone photos in real use (e.g. not in a review), give or take 100%, are of people. People move. Long exposures just lead to bad outcomes and blurred people.
I mean seriously search the net for Pixel 3 night mode. It's like the Suit Is Back. They're even using the same verbiage across them. And the uproarious nonsense about Google using AI to colourize is just...well a place like HN should just be chuckling at it.
Night Sight works perfectly fine on the Pixel 1.
> Why? Because 99.9999% of smartphone photos in real use (e.g. not in a review), give or take 100%, are of people. People move. Long exposures just lead to bad outcomes and blurred people.
I tested Night Sight with pictures of people and it also works fine in those cases. Even pictures takes with the front camera (without a tripod, etc.) look great.
Your Pixel 1 likely sets a ceiling on the exposure time. The results may be great to you, but I doubt they compare to a Pixel 3. And of course in all of these cases about these great photos, there are zero examples from any other devices. Just with and without on a Pixel device.
That's single frame long exposure, not many frames merged. And as you mention, unless you have a tripod or extremely good low light stabilization, in most cases you'll end up with a bad photo.
I would definitely like to see more with actual people in them, your point about humans moving is fair one, and I'd like to see how it handles them. That's where taking multiple shots and merging them vs a single super long shot would make a big difference, as you can smartly deblur things.
Any thoughts on why Apple, as the other leading phone maker with a heavy emphasis on camera quality, has not implemented anything like it? Not to discount the difficulty, but OIS aligned long exposures kind of seems like low hanging fruit. Instead, they keep trying to open the aperture more.
Stacking is quite the opposite of a "ghetto" version of a long exposure - it's the fundamental building block of being able to do the equivalent of a long exposure without its associated problems (motion blur from both camera and subject, high sensor noise if you turn up the gain, and over-saturating any bright spots).
Stacking is the de facto technique used for DSLR astrophotography for exactly these reasons -- see https://photographingspace.com/stacking-vs-single/
However, you're ignoring the _very substantial_ challenges of merging many exposures taken on a handheld camera. Image stabilization is great, but there's a lot of motion over, say, 1 second on a hand-held camera. Much more than the typical IS algorithm is designed to handle.
The techniques are non-trivial: http://graphics.stanford.edu/talks/seeinthedark-public-15sep...
There's a lot going on to accomplish this. It starts with the ability to do high-speed burst reads of raw data from the CCD (so that individual frames don't get motion blurred, and raw so you can process before you lose any fidelity by RGB conversion), and requires a lot of computational horsepower to perform alignment and do merging. I don't know what the Pixel's algorithms are, but merging of many images with hand-held camera motion benefits from state of the art results in applying CNNs to the problem, at least, from some of the results from Vladlen Koltun's group at Intel (who I'd put at the forefront of this, along with Marc Levoy's group at Google):
http://vladlen.info/publications/learning-see-dark/
I wouldn't be so quick to dismiss the technical meat behind state of the art low-light photography on cell phones.
You literally repeated exactly what I said image stacking was, yet lead off by claiming that I don't know what I'm talking about. Classic.
The goal of both is to achieve the exact same result -- more photons for a given pixel. Stacking is a necessary compromise under certain circumstances -- lack of sufficient stabilization, particularly noisy sensor or environment, etc.
Further, this implementation is clearly long exposures (note the blur rather than strobe).
If you read the slides from the Levoy talk I cited, you'll note that they explicitly choose to under-expose the individual exposures to minimize motion blur and blowout.
(Marc is now at Google continuing his work on computational photography, and his group contributes to many of the cool things you see on the Pixel series.)
But they don't. They don't in this example. Moving subjects are a blur. As an aside, of course stacked photo frames are underexposed because it wouldn't make much sense otherwise.
Computational photography can do interesting things and holds a tremendous amount of promise. However every single example that I can find of this mode -- across the many astroturfed pages -- show a longer exposure than what the stock app normally allows. And with that the requisite blurring of any moving subject.
Everything I can see you saying -- much if it agreeable, like the fact that long-exposure OIS makes a lot of what this technology currently does possible without it -- is simply handwaving away the fact that EIS-over-burst with OIS can achieve things that OIS cannot by itself.
It seems to me that it's patently true that EIS has some benefits, and those benefits can be realised over the top of OIS.
There's obviously still a fair limit to OIS. I have somewhat shaky hands and even using something like Olympus' top range 5-axis IBIS, which is the best I've ever seen, I can still only shoot at 1/10". What can EIS do with a burst of 3x 1/20" exposures? Probably counter for my shaking a bit, at least. (If not for subject movement, yet.)
I simply do not see why you're discounting this so heavily.
This all gets very reductionist, but EIS over a series of bursts is a bad alternative to OIS. It will be garbage in->garbage out. EIS with OIS, however, gives you the benefits of OIS, with the safety valve and "time travelling" effect of EIS (in that it can correct where OIS made the wrong presumption, like the beginning of a pan).
>and even using something like Olympus' top range 5-axis IBIS
The ability of OIS to counter movement is a function of the focal length. Your Olympus probably has a 75mm equivalent or higher lens, where a small degree of movement is a large subject skew. That smartphone probably has a 26-28mm equivalent lens. Small degrees of movement are much more correctable.
EIS is brilliant. OIS is better for small movements, but add EIS and it's great. Computational photography is brilliant. However Google has really, really been pouring out the snake oil for their Pixel line.
50mm equivalent in 135 terms, but yes, larger than 28. (I've since moved on to an X100F, but that's neither here nor there. :)
> EIS is brilliant. OIS is better for small movements, but add EIS and it's great. Computational photography is brilliant. However Google has really, really been pouring out the snake oil for their Pixel line.
This is what I was missing. It seemed you were arguing that computational photography is not capable of much, but you're more just pointing out that this computational photography is not doing much, despite Google's claims to the contrary.
I'd agree with you that this is not exactly revolutionary stuff.
If you stack the original exposures together, you'll get ghosting and not a blur. The natural-looking blur is a result of computation.
> of course stacked photo frames are underexposed, wouldn't make sense otherwise
Except it does make sense if you want to capture more shadow detail, this is how HDR images are made.
You're severely underestimating the amount of computation involved in getting these shots. These are all handheld, and as @dgacmu mentions can benefit from exposure bracketing which gives much better results than a single long exposure.
Of course you could already get similar shots from a good camera and technique - the fact these are handheld shots coming from a mobile device, and straight out with the camera is the impressive part.