Facebook’s AI unlocks the ability to search photos by what’s in them | Hacker News Reader

Facebook’s AI unlocks the ability to search photos by what’s in them

techcrunch.com

73 pointskosei66 comments

Google photos does this. I just opened Google photos and searched for "food" and aside from a couple pictures of some jack o lanterns (which are arguable, but I don't know anybody who eats theirs on Nov 1st), the results were spot on.

"Unlocks" might not be the right word here.

same in iOS, you can search your photos like that.

Actually, if you looked at the source on facebook, you could see photos being tagged with things like "one person, two people, smiles"

If you turn on VoiceOver on iOS (accessibility feature for the blind) and then tap on photos on the Photos app, it will read out whatever it found in that photo

The iOS version doesn't work very well for me. It often uses the location of when the picture was stored. If you receive a picture from someone showing Big Ben while you're in New York, it'll show the picture when you search for new york. So searching for places doesn't really work, searching in contents also seems to be much worse than Google.

Another point where Apple apparently lags behind in the AI game.

I found that both the Google and iOS versions could perform much the same sort of searches (e.g. "cat", "house") with roughly similar outcomes. However, Apple chooses to censor some searches (nothing illegal or dodgy, of course) and I could obtain no results for these despite having images which Google could find.

I think the idea here is to find things by analyzing the pictures directly as opposed to just searching contextual data like captions and link text. No idea if Google does that or not.

Edit: Didn't know Google Photos was a thing and assumed parent meant Google Images. Carry on.

I don't think you understood what GP said: he is using Google Photos, which searches his private collection. Not Google images.

I personally find Google Photos usually helpful and use it all the time to locate specific pictures. I do not caption them at all

I'm assuming that the "I am not a robot" image captchas are being used as a massive source of training data. I've notice there are several tiers of question:

* Select the images containing a street sign

* Select the squares in an image containing a street sign

* Draw a polygon around the sign

Someone better informed than me might be able to comment on how that fits in to deep learning strategies.

Almost certainly those are signs from Street View images.

That contributes to Google Maps - and accurate map data could have obvious value for self-driving cars.

Guy who invented it did a TED talk. Eventually used it to help translate books. Have noticed that it went to street addresses and now identifying stores, signs, and trees. So most likely for mapping applications.

Google has been leveraging people through captchas to do what computers are not good at for as long as google has been doing captchas.

If I search Google Photos for 'dog', the top seven hits are, in order: a fuzzy toy, a possum, another picture of the same possum, a cat, a pair of goats, a sheep, and I think it's a pheasant. The dog is #8.

Of course, if I search Google Photos for 'me', the top hit is a frog, so maybe it just hates me.

Update: not any more! Now the frog is #2. The top hit if I search for 'me' is the MV Isle of Lewis car ferry, photographed on the island of Barra in the Outer Hebrides.

Strange, I wonder why that is. I've got nothing but dogs on the full page. The first non-dog is a dog-collar on the 9th row. I don't get the ferry, either.

filter bubble probably: http://www.ted.com/talks/eli_pariser_beware_online_filter_bu...

I never used this feature until despite using Google photos extensively (I have ~60 GB of photos on there, granted that's 60 GB on my computer which isn't with the same compression/quality that Google uses). I have to say that I'm extremely impressed. it took multiple attempts to even find a mistake ("mountains", "city", "cat", "guitar", "food"). Although I was a little disappointed to find no results when I searched for "me". Being a frog is better than not existing at all.

You can add your name to a photo of yours (or any name to any photo). Google will then automatically find all of your photos. When you touch search look at the face of people that shows up below.

Interesting. I searched Google Photos for 'dog', and out of 5 results, one is my SO and I, and two are photos of my cats.

do you use google photos and/or drive regularly? Just curious if it is a training data set problem for "me"

I do use Google Photos - I set up automatic backup of the photos from my mobile phone.

It doesn't find anything for "me". I don't know what this classification is based on.

my mistake

That's something very different: this is for searching your own photos, not all those on the web. It's fairly easy to find dog photos on the web, since the web contains lots of websites with text saying something like "dog photos" and then lots of dog photos. But to find it in user-taken photos with no annotations is a relatively recent success (and is clearly not entirely there).

Wow. The search of my photos for dog is impressively bad. Everything from kids on all fours to monkeys with the rare photo of an actual dog mixed in lightly.

Google never gets any credit. I used Google Now before Siri exploded the world. I listened to a machine learning series, which started like: "... applications include Facebook's facial recognition, Amazon's product recommendations, Google's image search, and Apple's self-driving car." Apple's car? Google is the world king of ML, and you gave them image search?

Thank /you/ for giving us the credit!

I personally call Google 'God of the Internet', although AWS would be a pretty good competitor for that title.

As an addendum, I have two kids, and the google photos recognition rate for them is astonishing. They're ~2 years apart and google photos never confuses pictures of the older one with pictures of the younger one taken ~2 years later, for instance.

And in EU, the most useful feature face recognition, doesn't work because Google is afraid if privacy laws. After all the data I have given them suddenly the data I do want to give is kit possible.

Seriously, a 51.6MB "gif" that doesn't really add anything to the article?

Really off-topic, but: Are there (tech) news-sites which do not have this 10-100MB bloat with three js framweorks and gifs, ads and whatnot? I'd gladly pay for a minimalist news source...

There's AMP, if you don't mind trading one set of problems for another.

It's kind of a PIA, but I enjoy using umatrix and ublock origin. Allows me to only turn on the stuff that I want.

https://github.com/gorhill/uBlock/wiki/Per-site-switches#no-...

>Are there (tech) news-sites which do not have this 10-100MB bloat with three js framweorks and gifs, ads and whatnot?

The HackerNews comment section, with the added benefit of it being more informative.

You could do what I do and open them in Links/Lynx, a command-line browser. You only need the text, anyways, right?

i'll compare that with ublock/umatrix. thanks!

The page would have cost me 60 euros worth of data in Cuba. If I pressed refresh, because something didn't load? 120 euros.

I wonder, how does one survive on the "modern" internet in a place like that? I feel i would die a little inside. Seems the only reliable way would be to navigate with everything disabled, even images, and manually (or automatically) allow images that are of certain types (no gifs) / size (not too big) and etc.

I mean, that sounds so bad that i would argue you shouldn't even see the 50mb gif in that page, at all - ie it's your own fault. If your browsing "the web" with such a restrictive data plan you're playing russian roulette. It's only a matter of time before some JS heavy site or some blog with a gif "takes you down". Sure, sites waste a ton of data all over the place, but it's not their responsibility to "protect" you imo. You can't trust others to do that job. The browser vendors should be, or something.

Yeah, mobile phone apps which care about data are what you need to use. eg. use the gmail app instead of the gmail website. The opera browser is the best for expensive data general web browsing. Then you set up traffic limits.

The main way to protect yourself is to turn off the internet and enjoy life.

Apparently, there's a "weekly package" of videos, articles, and so forth that's used by 80% of Cubans. Distributed on memory sticks or external drives.

http://www.npr.org/sections/parallels/2015/01/05/374986028/l...

As we did before the "overly bloated a.k.a modern" web, which is pretty much as you described: disable all the superfluous (images, scripts, css, iframes, webrtc, etc.) and enable one by one those you actually want.

Better, faster, more secure browsing experience.

The issue is in the bloated websites to begin with. Which usually means that if one website is bloated, you just skip it.

It could have hidden data, or not.

I don't see how this is unlocking anything. Google has been doing the same for the past 5 years.

Google photos does this through their server.

Apple Photos does this through your phone, or your laptop. (Data from these two are not synced???)

Must be unlocking the same thing that Amazon Prime Photos is doing. I have to say that scanning photos automatically and allowing me to search by what is in the photo is the only solution to cope with the proliferation of photos, especially from camera phones. I still curate my better photos in Lightroom and publish them to Flickr albums. But that approach just isn't scalable for all the "normal" photos I am taking with the iPhone; so privacy concerns aside, I am happy for a machine to help me find the photos I am interested in.

If you want to see what Facebook tag your photos you can install this chrome extension: [Show Facebook Computer Vision Tags ] https://chrome.google.com/webstore/detail/show-facebook-comp...

In the recent times, for some other unrelated reasons, my browsing of FB pages and people's profiles was a bit slow from the network and download side (I always use a browser to visit Facebook, not the app). I noticed how accurate Facebook was in classifying photos when I saw the alt text captions for photos (before the image loaded). It was impressive, being able to say approximately how many people were there, whether a person was smiling, whether a photo was a selfie, whether a person was standing, whether a picture contained a landscape, trees and many other things.

It was also creepy that Facebook knew so much, because though I use Facebook for specific purposes and try to limit what I put there, I didn't imagine that images could convey so much information/metadata that would be useful to the underlying platform.

Recently on my phone I tried to copy and paste a picture from Facebook to my messaging app. Instead of copying the picture, it copied a text message that read "picture of dog and three people." I'm sure this is nothing new, but I'd never seen it before and thought it was interesting.

This reminds me, (very unrelated) a few years ago (almost 10), when I dragged and dropped an image from Wikipedia with Google Chrome to my Windows XP desktop, and to my surprise it actually saved it with a file name that is the description of the picture, not some arbitrarily chosen name or the Wikipedia file name. Sometimes it was really long and descriptive, like "A man wearing red and a woman wearing blue are playing soccer on a field blah blah blah".

I wondered where that came from, and I looked for that description on that image's page on Wikipedia and couldn't find the same description anywhere (keep in mind, I was a middle schooler or something and it could very well be the case that I overlooked something) I was a little spooked, and I thought it was some sort of hidden feature in Chrome that labeled images like that.

Does anyone know what was happening? It still bugs me to this day :P

AFAIR you can suggest a name for the file in response headers of HTTP request, so I'd suggest cURLing the image and looking at the response headers.

Could it have been the alt text for the image? I believe the grandparent comment was also pasting the alt text, since Facebook auto-generates alt text for blind users using ML (this is mentioned in the article).

now i know why facebook has been doin this. http://i.imgur.com/zPapJHM.png

I believe that was for accessibility reasons, ie screen readers.

but it also allows you to do the opposite things, like search for "sky" and find that image.

didn't know that, cool!

"alt" is what's read out by screen readers, or shown when an image fails to load (you'd see this a lot in a high latency connection).

Facebook's search is not useful. It's (obviously) segmented based on your own network, and generally useless for practical purposes. Perhaps law enforcement will find some use in this, but most people won't. Or at least, any utility beyond "Oh, remember that time I took a picture with that weird dog in it?"

Your power of imagination is truly inspirational.

I take it you haven't actually tried to put Facebook search to practical use. I used to run a nonprofit for combat veterans (since killed by California's AMT on all businesses including nonprofits) and used to pick up new candidates through word of mouth. I would then have to find these people often by name only. Facebook's graph search used to be okay for this purpose, but they've completely neutered all the utility out of their search over the years.

He's talking about its current public potential, not idealized potential.

For example go on facebook and use the search and tell me one interesing use case.

My worry: It will someday let agents scan your images for items of interest in an automated fashion.

Firearms. Flags. Knives. Religious symbols.

It's not to great a leap to imagine this happening in a very restrictive culture where possession of the wrong religious text could be considered criminal. Or a deck of playing cards. A cassette tape of music. Alcohol or Cigarettes. etc.

Never forget technology is just a tool which can be used by dark sides of humanity too.

The analysis of the images isn't meant for the users, it's so Facebook knows what's in them. Adding a user-facing interface is an easy-to-add toy for users and an attempt to recruit them into improving training data.

Thinking of just how difficult this is, I'm reminded of https://xkcd.com/1425/.

And see http://parkorbird.flickr.com/

Also http://birdsnap.com and http://merlin.allaboutbirds.org/photo-id/

To help with the implementation:

https://www.microsoft.com/cognitive-services/en-us/apis

https://www.ibm.com/watson/developercloud/services-catalog.h...

https://cloud.google.com/products/machine-learning/

We have a free app that uses AI to find unprofessional Facebook, Instagram, and Twitter photos used by over 50,000 college students:

https://www.renpup.com/

Reply on news.ycombinator.com