Learning to Predict Depth on the Pixel 3 Phones

Given that they built just one testing rig, and they use information like "commonly known object sizes" for depth, I wonder if they are using any transfer learning at all? I imagine whoever is bringing that rig around can't look at all objects, and transferring the knowledge of "common object sizes" and even what an object is can be hugely valuable for such a project.

I bet this is also a sneak peak into how powerful self-driving cars can be at Waymo. Not only do such cars have access to human data (sights, sounds), they can have way more sensors which can combine in a neural network to predict things way better than humans. For example, a NN could infer the chance a pedestrian might veer from the sidewalk into the streets by analyzing:

- The psychological state of a person using a high res camera on the face.

- Using chromographs to see if the pedestrian is drunk.

- Infrared cameras to see the person's temperature / if she's excited, etc.

- Limb modeling and likely walking paths.

I had this series of thoughts about smartphone camera "portrait mode" which were like this:

(1) Ha, pretty funny that we're spending all this human effort and R&D time to try to realistically simulate what is essentially a defect in camera lenses that makes them unable to have a wide enough focal plane to accurately represent what is in front of them.

(2) But wait, isn't that snobbish and a little shortsighted? (ha ha). What is a camera for anyway, is it making art or faithfully what it sees or something else? There's whole academic disciplines about this hmm okay.

(3) Well, no, I mean, a photo which has been selectively blurred like this has _less information_ than we started with, so it's objectively telling us less, and that's why it's funny, that we are spending so much time removing information.

(4) But maybe that's just the same as editing. Surely a good editor of texts takes a lot of information and removes information to make a better story; in the same way surely a "portrait mode" is really about figuring out what the most important parts are of a picture and selectively obscuring the rest, so that the 'good stuff' can shine through. That's important and hard; with an 85mm lens at f1.8 you'd do it by controlling focus carefully, but doing it after the fact is definitely interesting.

So okay, the reason this feature is interesting (and hard) is that software is trying to guess what parts of a picture are noise that should be gently deemphasized.

Now, it seems like the best way that software companies can think of to deemphasize unimportant parts of a picture is to mimic what a physical camera does with a portrait lens -- choose parts not on the same focal plane, pretend those parts were out of focus, then soften the image in the areas that "should have been" out of focus and blur them.

Now that they have shipped products that do that I'd be interested to see what the folks at Apple & Android do next. Do their teams still exist? Have they thought about how they would emphasize the important parts of a portrait if they weren't required to simulate the mechanical effect of a camera lens? What else could they do, maybe a little more creatively, to make portraits shine?

(And have they reinvented anything that Photoshop plugin makers have been doing for a decade?)

Interesting application of object detection/segmentation to make up for a weak depth map from the PDAF, but I have to wonder, if you're constraining yourself to portraits and are doing a pretty good job of detecting the person, I wonder how well you could do without the stereo depth data at all...