Hacker News Reader

Comment by phpnode | Hacker News Reader

Here's my guess at how it's done:

Redditor has facebook app installed on their smartphone (or just uses the website), sets status to "OMG wife is going into labour, at the hospital now". Facebook now knows roughly where redditor was at the specified time based on the ip, they can narrow this down further by looking at keywords in the status message and check it against a list of addresses in the local area and select the best match.

When the redditor comes to upload their photos days or weeks later, facebook just checks the photo timestamp against the user's location +/- X hours and makes a guess at where the photos were taken.

View on HN

It's probably a combination of techniques to be honest. They do have some unprecedented access to contextually heavy data, as well as an unusually large base of free labor to supervise the learning...

My creepy moment came when It correctly pointed out the exact location of this photo:

http://www.flickr.com/photos/40127665@N03/4788700749/in/set-...

I was in the middle of Costa Rica. For fear of roaming charges my phone was not on, and I never made a status update. The photo was taken on a D40 at the time, so no location exif data, and frankly the the picture is kind of generic.

My best guess is it used the other photos in the album to gain contextual information. For example this photo was in the same album:

http://www.flickr.com/photos/40127665@N03/4789329722/in/set-...

this, to me, would be extremely easy to recognize.

Simply not making a status update isn't enough. You'd have to never visit a page with a Facebook like button from a computer which you'd used ever to log in to Facebook with.

I'm yet to be convinced that this feature requires actually analysing image content at all. GeoIP and time stamps alone can provide a huge amount of context before you even start on things like mobile clients providing location data.

Both photos are in a public folder on Flickr called "Costa Rica".

the facebook album was named "D40 Pics", but it would be curious if it is cross referencing photos with flikr.

Or that species of insect only lives in Costa Rica.

Kidding! If it was that good, I really would be shocked.

The logic doesn't hold up. I uploaded a bunch of photos taken in Christchurch, New Zealand. The pics were over two years old and I now live in the UK and uploaded them from here. The photos were also taken on a cheap, point & shoot camera, not a smartphone.

but did you use facebook at all when you were in new zealand?

Indeed but when I posted the pics it was 2 years later from a totally different continent and no reference to the location on the photos.

right, but assuming the timestamp is on the photos, facebook still knows where you were at the time they were taken. The time you upload them is immaterial

In your opinion, how would Facebook know your location if you never logged in from a GPS-enabled smart phone? (and never approved an HTML5 geo permission request on your laptop).

That's the interesting question here.

well they can do a best guess based on the location of your IP, even if you only sign into facebook once while you're travelling. The really interesting question is: can they do this even if you have not signed into facebook at all on your travels, or for totally new accounts with no location history at all.

If they can, then they must be doing image recognition in some way. Perhaps someone who has more time / energy than me could try this by creating a new account and uploading some photos of recognisable objects (but without exif geo data).

Facebook knows more about me than I do. I'd be interested in their history file of my location.

> In your opinion, how would Facebook know your location if you never logged in from a GPS-enabled smart phone? (and never approved an HTML5 geo permission request on your laptop).

No real trick to that; if the user signs in from an IP leased to a Christchurch, NZ ISP on March 5th, 2007, then it's a safe bet that any photos taken that day were taken near Christchurch (and you get date taken from the EXIF data of virtually every camera on the planet).

When you were in NZ: [hh:mm dd/mm/yyyy, nz.ip.address] => [hh:mm dd/mm/yyyy, location]

When you uploaded the photos: [hh:mm dd/mm/yyyy, photo] + above data => [photo, location]

Guess: your friends also took pictures of the same events from similar angles and uploaded them, fb worked out their location from one of the other suggested methods, and then matched it to your photos through simple computer vision algorithms.

Or, you used facebook at some point while in that city, facebook recognised that your photo was of a baby just born, by comparing it with the millions of other photos tagged with "OMG MY BABY IS BEING BORN!," it knows that babies are born in hospitals, and it knows that there is only one hospital in that city, so it guesses that the photo was taken in that hospital.

It doesn't need to be taken on a smartphone. It's sufficient to open facebook on a smartphone only once (app or web) in roughly the same timeframe that the picture was taken. It can then look where one has been in the timeframe when the picture was taken by comparing the picture timestamp with the list of checkin ip addresses, and do a geolookup of the ip address against a location database.

Again, that doesn't align with my experience. The first smartphone I owned was when I arrived in the UK. In NZ, I was restricted to an old Nokia work phone.

Maybe you accessed facebook in a browser while in NZ? On a regular computer?

Also, does the location info purely say "Christchurch, New Zealand" or is it more granular? I'm guessing that when facebook only has sparse information about your location, as in your case, it "zooms out" to the city, state, country etc, but the more information it can collate, the more accurate the suggestion it gives is.

I reckon they use image recognition too. I have a gash digital camera that gets a lot of use; no exif data beyond camera make/model.

Anyway; in 2005 I went on a skiing trip to Chamonix. Pics were uploaded about a month later, no Facebook activity at the time (or while I was there). The other day FB asked me if they had been taken in Chamonix. So... something else used there at the very least.

How does Facebook use image recognition to recognize that an image of a baby on a blanket with no surroundings visible was taken inside a certain hospital? I doubt it.

Regarding your Chamonix images, perhaps your friends uploaded photos taken at roughly the same time, with geo-tags? Perhaps you are tagged in some of them? That would make for pretty simple logic:

- Facebook knows that you were at a certain place a certain time through the geo-tagging of those photos, as your user is tagged in one of them

- You upload photos taken at the same time as the photos you were tagged in

Conclusion: You must have taken those photos where the photos you were tagged in were taken.

Regarding your Chamonix images, perhaps your friends uploaded photos taken at roughly the same time, with geo-tags?

Nope. Family holiday. Of those there I am the only one with photos on Facebook - and in fact at the time was the only family member with a Facebook account :)

I am intrigued because if it is image recognition there isn't much for them to have gone on. But I am stumped for what else they could have used.

maybe Facebook just randomly guesses locations. those who get matches write comments on reddit wondering how the hell it works and those that get weird suggestions ignore them :)

Wow. Maybe they have a bunch of people looking at pictures, manually making suggestions? Google "similar images" search-like tool? It does sound scary if they really had nothing to go on other than the image data.

My photo locations seemed to be mainly based on the album title - I was impressed that it got both locations from "St Catharines and Niagara" correct.

I haven't checked to see if the individual photos are correct. However, at the time I didn't have a smart phone and I'm pretty sure my camera wouldn't provide much useful info.

UPDATE: I take it back: fb thinks I was at a football camp called St Catharines and Niagara! http://www.facebook.com/pages/St-Catharines-and-Niagara/1821...

I think that shows that the title is the most important indicator to fb.

Reply on news.ycombinator.com