Facebook Now Lets You Turn Any 2D Photo into a 3D Image Using AI | Hacker News Reader

Facebook Now Lets You Turn Any 2D Photo into a 3D Image Using AI

petapixel.com

100 pointsnotlukesky50 comments

Why not directly link https://ai.facebook.com/blog/-powered-by-ai-turning-any-2d-p...

Researchers at Adobe published similar work several months ago:

https://news.ycombinator.com/item?id=20978055

Interestingly, FB folks used a vastly simpler network architecture, but it probably still works because they have access to so much real data. In contrast, the Adobe paper had a fairly involved network with pretrained VGG features, possibly to compensate for only having synthetic data to train on.

It's also worth noting that FB doesn't need a lot of accuracy in the depth data for this to be "good enough" for a simple wobble effect on a phone.

It definitely doesn't need to be held to the same standards that would be held for a monocular depth estimator for, say, an autonomous vehicle or robot; in FB's case it doesn't matter whether the tree is 4m or 6m away, and really only matters that it's roughly behind the dog or cat or whatever.

Separately, depth estimators like this are also useful to fake the depth of field effects of large-aperture lenses, which I'm pretty sure is used on the latest Pixel and iPhone in their background-blurring "Portrait" mode; Facebook could quite possibly include that as an in-app feature to make such effects accessible to everyone with cheaper phones as well.

>If you’re more interested in how this impacts you, just know this: whether you want to convert “decades-old family photos” to 3D, or you want to convert your professional portraits into 3D-like creations, FB now lets you do that. You just need to upload it using the Facebook app on “an iPhone 7 or higher, or any recent midrange or better Android device.”

...aaaand that's the motivation for this feature - persuading people to upload more photos to Facebook.

lol, you say this like it's a revelation. Why would a company build a feature in their product and not expect people to use their product to use the feature.

There's a significant difference between "this works for all your current pictures on Facebook" and "you need to choose to upload some specific pictures on this specific app to use this feature".

The latter will get you a far better information on what people want to see turned into 3D.

The text is ambiguous, but I would interpret it as "Photos need to be on Facebook to do this, so those not already on[...]" rather than "This works only on Photos newly uploaded".

Why would they need to restrict the feature to new photos only to get that information? They could just use telemetry and see what photos people pick if they wanted to.

Like pytorch?

> ...aaaand that's the motivation for this feature

5 imaginary dollars says it's just something their AI team built and thought was cool so they rolled it out.

Wouldn't there need to be some business motivation to push that all the way through QA and localization and all the other steps before it hits production?

Zuck having an interest in 3D shit should do it

The business interest could be attracting talent by advertising their technologically interesting projects in the news

Call me silly but I am extremely certain Facebook is not exactly struggling to attract tech talent to the point that they'd build, test and deploy non-trivial features just to do so.

Maybe the reason they're not struggling now is because they already have a history of doing things like this.

Or maybe it's being one of the biggest companies in the industry and having hundreds of billions of dollars to throw behind compensation and other perks.

Nah, you're right, they put entire teams to work building and maintaining user-facing features so they can add a couple thousand more applicants to the hundreds of thousands they already get.

That's also true of, for example, SAP, Oracle, Accenture, etc., but would you say they have the same chance at attracting talent as Facebook?

Why would it need an "iPhone 7 or above" when the depth map is calculated on FB's servers?

3D Image ≈ Lovecraftian monstrosity. I've seen a few examples of this, and the results are mixed in a rather entertaining way!

To be fair, raw data from "real" stereoscopic approaches looks fairly similar. Definitely a good result, but certainly something I have seen before in a comparable quality. Hard to judge from the examples though.

Anyone remember an iPhone app canned Seene (I think) that did similar about 7 years ago? It lacked the AI aspect but essentially you took several photos from different angles and it created some form of depth map with the central image as a texture (if I remember correctly) I was amazed by this at the time.

Photogrammetry is a commoditized set of algorithms and workflows to do this.

https://en.wikipedia.org/wiki/Photogrammetry

There is a good introduction to production use in this GDC presentation from DICE:

https://www.gdcvault.com/play/1023272/Photogrammetry-and-Sta...

Examples of applications that are usable for individuals include:

https://www.3dflow.net/3df-zephyr-pro-3d-models-from-photos/

https://www.agisoft.com/

https://www.capturingreality.com/

And free/open source/academic:

https://colmap.github.io/

http://ccwu.me/vsfm/

https://alicevision.org/

https://get.display.land is also pretty good. It did a passable job of our local statue[1] in about 3 minutes of capturing + [some time of their cloud processing].

(Now it's not in the same ballpark as the capture I made with 5 minutes of taking ~90 photos and the ~20 hours of Meshroom processing but that's understandable.)

[1] https://www.display.land/s/4QeZc8gT77beNLx44OdvDT

The difference here is that they (FB) are able to create 3D model based on only one photo. If you have multiple photos from different angles, you can just use geometric relationships to figure out where the points lie in 3D space. However, with only one picture, you need to "guess" (using ML), as there's not enough data for proper 3D reconstruction.

I remember playing with effects like this ~8 years ago. My first instinct would be to laugh at the "AI" part, but I'm guessing the point of the exercise isn't the parallax effect - the point is that they've actually managed to get a neural network to do that.

I think the point is that it's far better than anything you may have been doing 8 years ago?

The picture of Trevi Fountain has a very complicated 3D shape, almost none of which was inferred correctly.

The article was singing its praises, meanwhile I was trying to figure out which parts the system did get right...

It's entertaining at least :)

Yeah, and that's the one that was good enough to into facebook's advertising material.

I would call that 2.1D the .1 adds parallelax effects

It's 3D depth estimation from a 2D image. There is no such thing as 2.1D, 2.5D unless you are either a salesperson or a theroetical mathematician.

Parallax effects are not 1/10 of a dimension in any sense of the word.

Obviously it does not literally add 1/10 of a dimension. However, saying "2.1 dimensional" gives people the idea of "oh, it is still mostly the same as a 2 dimensional thing, but is a little bit closer (in some sense) to being 3 dimensional.".

In commercial motion graphics/animation, 2D images animated in 3D spade is widely referred to as “2.5D”

"convoluted neural network" :)

As someone who works with them, they certainly seem like that sometimes :)

Only in TensorFlow...

Where "3D image" = amorphous blob.

I really want background replacement to be totally seamless. We've got to be close to that with this technology.

How many people like watching 3D movies at home or at a movie theater? Sometimes old fashioned 2D images are just fine. The content, story and artistic aspects of the image is what matters, not some cool half-baked 3d trick that is... frankly a distraction.

I don't mean to criticize the research and engineering that went into doing something like this - it truly remarkable. I refer to the application possibilities that we might see in the future - imagine the web full of these 3d images, I think that would be a pretty noisy and distracting place.

I think the 3D effect is just a conversation starter. The purpose of this is to add couple new slider options to Lightroom.

If you extrapolate where social media apps are taking us, first it was to "keep in touch" (however superficial) with people, even when it's a tweet that "they're pooping" or an IG of what they're having for lunch. And then came Periscope livestreams and Snapchat stories (stolen by Instagram and Facebook), so you can see a live video of what your friends are doing. FB bought Oculus, I wonder if their fantasy of the future is for realtime immersive sharing of experiences, your friend is on top of the Eiffel tower, you put on your goggles, and boom, you're there with them with a 360 degree video.

At least that's the technological side, personally I think this simulacra of friendship/socializing is terrible.

You should watch some of sucks F8 keynotes. He lays out basically what you're saying

Would this work for converting 2D panorama photos to 3D for VR purposes, essentially creating a fake stereoscopic photo?

Hey! Here is a novel way to get people to submit their photos to Facebook.

I wonder if this could be used to improve background removal tools.

Can anyone recommend a good open source version of this?

Never tried it myself, but this might be an alternative - https://github.com/sniklaus/3d-ken-burns

I don't need Facebook's permission to do that.

Why did we need this ?

Are we now going to remake all pictures into """3D""" and leave the boring 2D pictures in dust ?

They can make this very cool yes, but still isn't enough for me to enter Facebook. They should have more money for privacy.

Reply on news.ycombinator.com