The iPhone 11 Camera, Part 1: A Completely New Camera
blog.halide.cam
blog.halide.cam
Show me the lenses. Put the coatings under an electron microscope. Cut the sensor in half, and measure the depth of the electron wells.
Then on the electronics side, hook up a logic analyzer. Find the speed of those ADC's. Dump the default register values.
And on the software side, feed in manufactured frames to find out exactly how the HDR algorithm works. Use high speed strobe lights to find out the exact shutter timings. Figure out if it uses the gyroscope or optical flow for frame stacking, and how many frames does it even use? Publish the bit depths and colour spaces of intermediate image pipeline stages.
These 'technical' reports which really boil down to "we took some photos and they're quite crisp" really disappoint me.
https://www.techinsights.com/products/def-1909-803
They'll charge you however.
The camera has usually been my primary motivation for upgrading, and I'm tempted. I'm still sporting a 7+ with broken lenses. Which has meant mostly terrible cell-phone pics that occasionally due to a happy accident look like a "Holga" picture.
The upside is that I went back to really actively using my DSLR, and I'm not sure anything really compares still. However, this article was pretty compelling.
I think the title is accurate – these pictures no longer look like 'iPhone' pictures, which was a kind of brand identity more than a technical constraint. This feels like a refreshing shift. That said, they have a look. They remind me of Andreas Gursky photos – hyper realistic, super detailed, somewhat matte.
For example Portrait Lighting can run on hardware without the neural engine, if you hack the exif to say it was shot on a newer device. However, this is just when editing. There's no way it would be capable of the realtime preview, and casual users rely on the preview to make the feature useful at all.
Actually, it would confuse a ton of users. We added portrait-mode support for non-humans on the iPhone XR. The hardware doesn't provide realtime depth information, so we can only apply the effect after you've taken the photo. We frequently receive support requests telling us that the feature doesn't work.
I can't speak to why Deep Fusion isn't coming to old devices, since it's an offline process. I mean, I could speculate that if it relies on multi-cam capture to capture multiple images, that's strenuous API that isn't supported on hardware earlier than the XS.
The cynical perspective is that Apple caps features to the latest hardware to sell new hardware. I'd say it's just as likely that Apple looks at the new hardware coming down the pipeline and plans features that push the hardware to its limit.
Great, then why doesn’t it work on my XS?
Also, if you're going to share a photo, you're likely going to do it the same day you took it.
About 60-75% of the time I take photos, I share them soon after or within a few hours. Having to wait to let them be processed overnight would be obnoxious.
And it's not particularly helpful to compare people taking photos on their phone to the DSLR world. (Coming from someone with a nikon d200, d300 sitting on his shelf). Totally different use cases, totally different usage expectations.
I know Google made their night model generally available, but I don’t think it depends on specific custom hardware.
I wish they would, but that doesn't fit Apple's typical MO.
If anything, it ensured my next phone would be a Samsung again (and it was, I went Note 10), so I think it's still a good upselling move to do so.
On that note, how far they've come since the Note 4 terrible TERRIBLE result in low light, that phone was top notch for its time but in low light it took several seconds for a single shot, focus sucked, grainy as hell ... Seeing a 9/10th gen phone instantly take those shots, with almost no grain and a lot of back details properly lighted is very impressive to me. I understand how it works in theory, but there is still a "wow I can't believe my phone pulls that off" factor for me.
Android vendors had (have?) a horrible history on this...
My SE has really been struggling with iOS 13. A lot of apps don’t load and things are ssssslow.
Of course one could argue that the iPhone 8/X, XS, XR buyers did finance the R&D for it. Each year Apple has beat their neural engine drum but it wasn't until this year that they really introduced features that took great advantage of it. Sure, they used it for FaceID, but I'll wager it didn't really tax the silicon much.
Sure it would be nice to get some special new features, but as mentioned, that also doesn't suit how Apple operates when it comes to new features. And that's quite alright.
Apple, or any business trying to make money?
I picked up an iPhone 11 pro last week, upgrading from a 7. The camera really is great. What I keep thinking is that I’m finding it creepy. I’m not a luddite by any means, I’m all for innovation….I shoot with a Sony mirrorless camera as my main everyday, that felt to me like a massive change.
I think what gets to me is I don’t know exactly what the iPhone camera is doing. When I press the “shutter button,” what exactly is happening? How many shots is it taking? Is it shooting on all three lenses? Is it recording audio even when I have Live Photo turned off? If I think I’m intentionally leaving something I don’t want to record out of the frame, how can I be sure that one of the other lenses isn’t also recording and including it? I just don’t feel like I’m sure about any of it anymore, with all the processing I hear about it doing, layering multiple shots together, shooting outside the frame. The tech is cool and all and I’m not prone to paranoia, but I just keep wondering.
I felt EXACTLY like that using my first mac coming from unix. Apple hides things and I never quite knew what was going on under the hood.
But now it is like 1000x that. Run "ps -ef" on the command line from macos and wonder. I have 311 processes running. What is dprivacyd or com.apple.geod or photoanalysisd or ProtectedCloudKeySyncing?
With photography I feel your pain. Everything is amazing, but at the same time really dumbed down. You don't need to know how Ansel Adams zone system, but... you don't know exactly what is going on and you have little control. Sometimes you need white balance or manual exposure or manual focus or control over the shutter speed, or a real shutter button.
sigh.
Everything else...could probably rock my 7+ for another year or two. Maybe do a battery replacement.
The iPhone 11 pro blew me away with the night mode photography, and the simulated bokeh that can be modified in post because of the depth camera is downright genius. And the most important feature, it’s always with me, unlike the DSLR.
I’ve been using my wife’s hand-me-down phones for a decade now. This was the first phone I bought for myself, and really it was a camera that has a phone feature, not the other way around.
2014: https://ai.googleblog.com/2014/10/hdr-low-light-and-high-dyn...
2017: https://ai.googleblog.com/2017/04/experimental-nighttime-pho...
Really makes you wonder what could be achieved by combining Google's computational photography algorithms with a bigger sensor and high quality optics. I'm probably not going to buy another high end camera, until I see something like that on the market.
But indeed, camera manufacturers are way way way behind. The best they can do at the moment is find faces on a picture (and still, not that well except on Sony cameras). This is disappointing to say the least.
The Zeiss ZX1 runs Android, but I don't know how much computational photography it provides, since all the reviews about it only emphasize the fact that it can run Lightroom (which sounds completely useless).
After comparing them, I'm convinced that it's only a matter of time until phones are going to catch up to full frame cameras. Software these days is more important than the raw hardware it seems...
Having recently purchased both the latest RX100 and the 11 pro, I both agree and disagree. For auto mode in certain conditions, when viewed at significantly scaled down resolutions, yes. To if you're posting to instagram or facebook or whatever, this is probably true.
Manual mode on the RX100 vs the 11 pro, even when using custom apps that expose a lot more of the camera settings, the RX100 starts to pull away. If you're doing any work with larger image sizes (and by larger I mean using a computer or TV screen instead of a phone), even more so. They're honestly not even close at that point.
Meanwhile, on the full frame side of things, the A7 line of cameras has been advancing picture quality at a rate as fast as anything in the iPhone, and just absolutely clobbers it. There's physics in play here, and I'm skeptical that software or processing power is going to increase the picture quality that much, and I have no idea how you would match things like super telephoto lenses in a smartphone form factor.
I do really love the iPhone when I just need to take a quick shot, though. All the computational stuff means that I'm almost always guaranteed to get a usable photo out of it in any condition.
What could a DSLR with iPhone-level software achieve?
The problems for camera manufacturers around this, though are:
1. Their market is incredibly, loudly regressive about a lot of this stuff. A noisy chunk of the photography market is really hostile to workflows that don't mimic hundred year old darkroom processes. Doing in-camera is an abomination. Automation is an abomination, etc.
2. Building the compute in to the camera is non-trivial. You've already got a lot of compute power focusing on running the complexities of things like continuous AF tracking (e.g. on top-end Olympus that's 4 of the 8 cores available). A lot of compute budget is used for things phone cameras are rubbish at. At some point things like heat become a problem.
3. Data volumes are hard: Olympus are "only" dealing with a 20 MPix image. Sony are dealing with 24 - 60 MPix images. Olympus do that at up to 60 frames/s (20 with AF), and to read the data within the constraints of shutter speeds of 1/16000 s. That is... a lot of data compared to the relatively modest rates on an iPhone. Oh, and while iPhone users are generally OK with a bit of lag, DSLR users get really pissy if you're making them wait for that. Latency needs to be very low.
4. Physics is hard: A Sony sensor is a 24 x 35 mm hunk of silicon. Quite apart from the challenges of the data volume, reading a chunk of CMOS that large has been a challenge. Sony have done a lot of clever things to work around those limits, but still... (Olympus have been able to do high frame rates longer than anyone else in part because their sensors are smaller)
And, if I ever need zoom (like in nature photography), phones just can't compete with a 300mm lens.
If you have a Canon camera, check out the CHDK
Now if they can just make it smaller next year.
EDIT - I should mention that the UWB radios might be cool once there's a significant feature using it. In terms of future-proofness, it's a handy thing to get even if it didn't launch with any important functionality.
With 11 Pro Max, I charge once every 1.5-2 days with regular usage. This is truly amazing to me, because I never have to worry about my battery dying midway through the day, and it feels extremely freeing.
But, if you are taking lots of pictures or about 15 mins of video, the battery seems to drain quickly, by about 45% as I was doing today.
The fast charge with the 18 W adapter is also very helpful.
On my S10 I now have a cheap $5 charging pad at the office, and I never really think about charging anymore when I get home.
Only on vacation trips I still have to budget my battery.
The 11 Pro's camera is so much better. The edge detection in Portrait Mode is better on the 11 Pro (I think it's the hardware, not just iOS 13). Night Mode is unbelievable. Video has improved also (I think this was mostly due to XS upgrades, but there are other 11 Pro benefits that I mostly won't use, like simultaneously capturing video from multiple cameras). I haven't done the .2 update yet, but the deep fusion stuff is pretty cool from what I'm seeing others post online.
I think folks like Brian X. Chen (NYT), who say this is an optional upgrade if your iPhone is less than 5 years old have it wrong. I can definitely see skipping this if you're on an XS or X. But 8 or older? It's a very noticeable upgrade. And if you're not on a Plus (i.e., capable of Portrait Mode), it's a huge upgrade if you use your camera much. And that's before getting to all the other features like wireless charging and the UWB chip for enhanced tracking. Oh, and the battery is noticeably better than my 7+, which had great battery life (and was warranty-swapped by Apple a year or two ago, so the battery was not a full 3 years old when I upgraded to the 11 Pro). The 11 Pro is the first iPhone I've had that I do not need to charge overnight. I just charge it periodically during the day, once every 36-48 hrs.
My work iPhone is a 6, so it's LED, and even though I keep its screen on eight hours a day, there's no burn-in.
Where do you see burn in? Light or dark screens?
Any time there is an area of flat color, or gradient color it really shows.
This one from the article is a pretty good example. It looks like a photorealistic painting when you zoom in, and not just on the sky, like they point out in the article - the whole thing does.
I'm really impressed with the 11 pro for stuff that gets viewed on the same medium it was created in, but the trade offs are very apparent when you look at things on a computer monitor.
From the article:
> If we distort the contrast in an image, we can bring out the ‘watercolor artifacts’ in the clouds
I think you missed that sentence. You posted the "distort the contrast" examples, which were hand edited to exaggerate any artifacts in the photo the iPhone had output.
This is the photo that the iPhone gave the photographer: https://miro.medium.com/max/1800/1*oZgBZGHyxwMevHKiGfATiQ.jp...
I think you missed part of the subtitle. The one on the left has had contrast adjusted, the one on the right has not and matches the one you posted. The watercolor artifacts are just as visible on your link as they are on the right half of the picture I posted.
I find them incredibly noticeable, even on your link.
I wonder if there’s a line where copyright could attach to the developer who wrote the code instead of the thumb owner.
There was also a photographer who was in Britain for some show. He wanted to do some photography but wasn’t allowed as he was only there as a tourist and didn’t have a work permit, so he handed his camera to his daughter. He then set up an exhibit with her work. I’m not sure who “owns the rights” but he assigned them to her so as not run afoul of his visa.
[1]https://en.m.wikipedia.org/wiki/Nussenzweig_v._DiCorcia#Laws...
https://en.wikipedia.org/wiki/Monkey_selfie_copyright_disput...
Sensor fusion has been a hot area of research in robotics for at least a decade. Why do you feel it gets no attention?
for example, photoshop's liquify feature can now directly isolate eyes, noses, lips, cheek bones, facial structures etc without you needing to be good at using the liquify brushes
but ios is now doing this kind of thing automatically and integrates the editing right there either in the photo by default or in an integrated editing app
As far as technical features, things get really interesting with an iPhone 6S and later. That's when Apple added RAW support, which requires a third-party app. On depth capable devices (7 Plus, 8 Plus, X/XS/11), we have detailed depth visualizations.
Anyways, we're generally forward looking, we'd be crazy not to serve most of the folks out there running older phones.
It obviously doesn't have the features related to multiple lenses and smart HDR, but it is my go to camera app.
Google & Apple are digitally altering photos so much so to “enhance them”, couldn’t it be agrued the evidence of a crime photo has been tempered due to computational photography.
Definition from OED: “interfere with (something) in order to cause damage or make unauthorized alterations.”
The “in order to” seems to imply intent. Not sure if there’s a different legal definition however.
Plus none of these modifications really alter the content of the photo.