Facebook’s automatic alt-text for images | Hacker News Reader

Facebook’s automatic alt-text for images

facebook.com

196 pointsboyter77 comments

Accurate, automatic descriptions of snapshots of peoples lives?? This is surely sending shock waves down the data mining community. Additionally Facebook are saying this was built as an aid for blind people, but this is surely just a cover for being able to take targeted ads to the next level.

This is surely sending shock waves down the data mining community.

Not really. From a technical point of view there is nothing impressive about this, except that they spend the time and money to collect all the training data. Also that it says "this image may contain:" tells you something about how accurate this actually is.

Tensorflow + Inception is about this accurate. For high level categories and popular items it's pretty good, but if you want to dig deeper (e.g. identify the type of animal in a picture) it'll need more training.

Google Photos is a lot smarter in that it can identify what a set of photos contains. As an example I noticed all pictures of my graduation (which had no metadata saying it was that) have been grouped into a 'Graduation' album. It's also kind of scary...

Did you have EXIF data on your pictures? Similar location + even a single picture with an academic cap could easily provide enough information for categorization...

Indeed, in this presentation given by Geoff Hinton[0] (video uploaded 13 oct. 2015), a similar system to this is explained very simply. And by the time this was recorded, and how it was presented, I get the feeling it was not ground-breaking or cutting-edge -- but still a very good example to show.

[0] https://www.youtube.com/watch?v=IcOMKXAw5VA

Not my point. What isn't available to everyone is the sheer amount of real life images that Facebook has access to, which people are uploading of their lives as we speak. You could get an unprecedented view of almost anyone's life from that, seeing as users have past the 1 billion mark, and it's all in the hands of a company who sells your data for profit.

Your basic demographics and "likes" are already enough to do all the targeting necessary.

If this is about the general privacy issue then it's back to the same old thing: don't share things that you want to keep private. Of course people's actions prove that they get more value out of sharing this content then is worth the privacy of not doing so.

Again, not my point. Social network sites are built off image sharing, that before computers did not understand. Now that they do, we are entered an unprecedented age of surveillance. Private is only private to humans, the Facebook machine can look at any photos it wants, and build up a profile for anyone, their friends, make predictions of behavior and so on.

It's not the tech that is bad, it is the fact it is in the hands of for-profit companies, who are willing to sell your data just to make their bottom line.

Then what is your point? manigandham literally adressed your concern; basic demographics, likes, and other sources for content are far better for targeting than what a picture could potentially contain, and that can be done with closer to 100% accuracy since a computer doesn't have to "guess" what the content of an image is, it already knows this content from the text in a status, or the location metadata associated with a post, or the people you're tagged with. This is incredibly inaccurate when compared with all those sources.

Posted images contain far more information that what is posted in text. Scroll through your feed, people upload images then write a header that adds to the scene. It doesn't describe it. There is so so much more information to gain from parsing images, I can't believe you can't see that, and can only assume you aren't thinking this through properly.

No, I am thinking through this properly, you're just being overly paranoid. A computer ML system right now will not be able to, with a high degree of accuracy, know enough about the image to provide targeted advertisements. They even showed samples of descriptions; they're vague and general, not specific.

The description might be "two people, outdoors, at beach" while the content could be "Awesome beach day today with my best buddy!" with X tagged in the image, geolocated to "Y beach" (either through your phone/camera's geolocation feature, or the poster specifically tagging a location which is something Facebook encourages), posted at Z time. Which do you honestly believe is going to provide more information for targeting? Hell, even if the content is "It's a beautiful day out!", you'd still have far more information from the metadata (tags, location, time) than the vague description the auto-generator could possibly provide.

> This is surely sending shock waves down the data mining community.

Tech keeps getting better and Facebook keeps getting more data and putting it together results in better insights. There are several other major companies and government agencies who can derive just as much, if not more, insights about people through data. This is just natural evolution, not a major revolution.

> but this is surely just a cover for being able to take targeted ads to the next level.

As someone who has built 3 adtech companies, this image info is nowhere near as good as the current posts, relationships and likes that users have already entered. Maybe someday it will be able to accurately know what's actually in the image and also derive some intention and relations out of it, but today it's nothing special at all.

> You could get an unprecedented view of almost anyone's life from that, seeing as users have past the 1 billion mark, and it's all in the hands of a company who sells your data for profit.

> It's not the tech that is bad, it is the fact it is in the hands of for-profit companies, who are willing to sell your data just to make their bottom line.

So are they not allowed to use the data? They are a business and will use everything they can to make their system as good as possible. Whether users care or not is up to them but it's clear that people find plenty of value in Facebook to continue using it.

If you're just stating that the possible surveillance capabilities are increasing then I think this is pretty well known...?

It is not currently, but the point I am making is the tech is coming in leaps and bounds, and I expect the next 5-10 years we are going to see something we haven't seen before; computers parsing semantic information from images, something that once was just an array of pixels to them. I point you to this TED talk where they can already describe scenes [1] (skip towards the end).

> So are they not allowed to use the data?

Being able to pull semantic knowledge from data changes the game. I expect in the next 5-10 years laws are going to have to come out to stop companies going too far with this. Sadly, laws are only really introduced after something goes horribly wrong, so we've got that to look forward to too.

> If you're just stating that the possible surveillance capabilities are increasing then I think this is pretty well known...?

Privacy advocates are a minority. My friends are almost all non technical and don't even think about it. That's because this is new ground, and there haven't been severe repercussions yet. But they are coming.

[1] https://www.youtube.com/watch?v=40riCqvRoMs

I kind of want to take this at face-value and say how great it is that they're making visually-impaired viewers a priority.

If that was the case wouldn't they choose to simply not announce anything?

I'm not ruling out that they might use it for targeting eventually, but if this was solely done as a cover it would be the equivalent of a terrorist entering an airport shouting out "My suitcase is just really heavy, I do not have a bomb in it at all!"

> If that was the case wouldn't they choose to simply not announce anything?

People would find out within a few days and out would come the pitchforks. Here facebook preempts the conversation and sets the tone, now worriers will be met with "Do you not want blind people to enjoy the web? Why do you hate them so?"

Do elaborate how people would find out if Facebook simply used this data internally.

Might take a big longer, but I expect people running and testing ads would discover correlations with image posts.

Facebook was already using this data internally. You just didn't know they could do it until now.

The goal is to normalize people to yet another type of surveillance and data mining.

Could be worse. Google has me training AI to do the same thing every time I choose all the pictures with train locomotives in a reCaptcha.

I always wondered how many captchas were passthroughs to other captcha systems, using me as a mechanical turk component.

Definitely exists. I remember seeing a post a few years ago about a porn site that had "Type in these characters to get the next image" and had people solving captchas for them.

For a captcha to function properly, they already need to have this data I guess.

reCAPTCHA is famous for having 2 tasks. This lets them use 1 for training.

https://en.wikipedia.org/wiki/ReCAPTCHA#Criticism

Not only this, but given how much google tracks you, google has a good enough idea about whether you are a bot already, hence the "captcha" where you just press a button.

There are times they could give you that button but already knowing you are human they use you to train their AI.

I wouldn't be surprised if they also score you against how well you train AI, and give better trainers more captchas.

Google has so much data, any time you think "I wish we could do X, but we'd never get enough data", it's an opportunity that Google can move on but the rest of the world can't. That's another argument for breaking up monopolies.

> it's an opportunity that Google can move on but the rest of the world can't. That's another argument for breaking up monopolies.

Wouldn't this mean that it would be an opportunity that /nobody/ could move on?

Google, Facebook, twitter .... Are all doing this in their backend. If you are worried about it. Don't use their services.

The problem I see here is that now they are giving this data to spammers.

> If you are worried about it. Don't use their services.

I don't use their services but apparently I'm still tagged in Facebook photos. I never accepted ToS permitting that.

If there's a picture of you floating around in public it's not illegal or immoral for them to say who's in it.

It is immoral for them to collect enough infos about me to be able to tell who I am from a picture, and then use it, while I do not use their service.

Why's that? Assuming all the info is public, they've done what any person could reasonably do.

Put another way, with absolutely no snark or malice intended: Information about you is not necessarily owned by you.

While I kind of agree with you, I don't think it is fair to automatically assume that just because an automatic computer system is doing the same thing that a human could do, it can't fundamentally change what that thing means. Things change when something can be done automatically and at great scale.

Take, for instance, license plate scanning. Of course, anyone can see your license plate when you are driving around town, it is public information.

So clearly it is fine if machines scan license plates and keep records of every license plate that crosses a bridge. A team of people could sit on the bridge and do the same thing, so no big deal.

So clearly it is fine if we install the license plate scanners on every street corner in the city, and scan every plate they see.

And it should be totally fine to put that data on a publicly searchable real-time database, so that I can input any license plate number and see immediately where that car is.

These different scenarios might seem like merely matters of degree on the same bit of info, but what it means for us changes immensely as it becomes easier and faster to do it.

I agree that the license plate situation is a concern, but banning the scanners is the wrong fix. I could organize a group of friends to all point cameras out our windows and do the same thing, without the help of government (and in fact, have sort of considered doing so just as experiment, to prove how trivial it would be). The privacy incursion occurs when the government requires you to attach an easily recordable unique public identifier to your vehicle, not when they document sightings of the identifier. Once you put the license plates on the cars, you've lost already -- anybody could be looking, and you'd never know; it'd be basically impossible to regulate.

Not to say I don't think there should be license plates; obviously there's a public safety argument for having them, and that argument has to be counterbalanced against the privacy concerns (and perhaps rebalanced as technologies advance and change the relative difficulty of some kinds of privacy-invading tasks). But focusing on the scanner is pointless. More broadly: acknowledging that people trivially have access to tons of data that, when taken together, damages privacy, and trying to fix the privacy problem by telling them they're not allowed to look at it, is never going to end well. If the problem is solvable at all, it's by controlling access to the data in the first place.

I wasn't arguing that we should ban any technology or telling people not to look at them.

What I was trying to say is that being able to do something fast and at scale is not just a progressive improvement from being able to do them slowly and by hand; it is actually fundamentally changing what the thing means. What we decide to do with this fact is another question.

Heh, I figured this would come up, but I think where we disagree is in degree, rather than action.

In your above scenario, I'd start calling for the metaphorical heads of politicians somewhere between "installing scanners on every corner" and "public searchable real time database".

The example we're speaking of here, Facebook has collected the information that the GP poster exists and looks like this. Already publicly and digitally accessible information. Nobody's privacy has been violated by saying that person X exists and looks like Y, nor could that information be used against them. I certainly hope you understand that a live database of car positions is fundamentally different...

On top of that, the genie was unbottled ages ago. You may be able to make some kind of privacy argument re: the plate scanners now, but as the tech advances more and becomes cheaper and it becomes trivial for anyone to do the same thing (I recall an article about someone who set up a scanner in his front yard with some open source toolkits and relatively cheap gear), the arguments that "anyone can do it, except these people, because reasons" start sounding more and more arbitrary and out of touch.

But I don't have access to all the photos the Facebook neural net does, because of private profiles and the fact I'm not friends with everyone. So really now the machine has far more information that I do, and it can make predictions I can't. Like, where anyone is likely to be at any time, by analyzing photos and matching locations between those photos. Think of what you could do when you can map out the daily lives of millions of people?

We haven't seen the full force of what this means yet as it's still early days, but with for-profit companies behind the wheel like Facebook who make money from selling your information, this is an unprecedented level of big brother-esque invasion.

I think you are quite wrong about the "Nobody's privacy is violated by saying that person X exists and looks like Y."

For one thing, you are leaving out the part that says "and doing Z". There have been lots of cases of people having tagged photos be used against them. While you might be fine with the cases of someone being tagged at a baseball game when they said they were sick from work, but what about when someone is tagged at a gay pride parade and they live in a homophobic area? Or being tagged in a picture of a protest when they live in an oppressive country?

Can I ask you to elaborate on how and why? I instinctively agree with you, I think, but I don't know how to explain it.

How can someone tag you if you don't even have a FB profile?

I just tried to tag the fictional "Jimmy Andersefsdfsdfsd" ad it doesn't work....

Well if 98% of your images are pictures of cats, you don't really have to solve the hard general problem.

You do if you want high accuracy on stuff other than cats.

This has been possible for, at a minimum, three years. There's an effort gap between building an ad targeting version and building a blind enabling version.

The only new thing here is that Facebook released it to the public.

I always find Facebook's example feed to be funny since it's a completely unrealistic depiction of what their site actually is for most users.

Just quickly looking at the top few posts in my feed, I see someone celebrating their two year friendship with someone I don't know, one person sharing a link to a new airplane, four people sharing videos, one person liking a sponsored video, and finally one person updating their profile picture.

I wish I could see actual updates from people instead of being kept abreast as to what piece of third party content they've liked at some point in time, what third party content they're sharing, etc.

How is this unrealistic? This is exactly the kind of stuff my feed shows. The people you follow probably aren't posting any updates (that you like to see)... and sharing content is an update, that's what that person decided to post.

Your feed is what you make it.

Everytime I see someone complain that their feed is full of memes, junk, and chain posts, I think 'Maybe you should just remove that person from your Facebook?'.

> I wish I could see actual updates from people instead of being kept abreast as to what piece of third party content they've liked at some point in time, what third party content they're sharing, etc.

This nicely captures the problem. Facebook (and probably most social media systems - I'm looking at you LinkedIn) are primarily interested in keeping you up to date with Facebook via the medium that is your human relationships. So Facebook wants you to know what your friends did on Facebook today so that you might do that same Facebook thing. This increases Facebook interactions which in turn become more information to propagate to others on Facebook. In the limit there is no need for Facebook because all anyone is ever doing is Facebook.

The first thing I thought was, "I wonder how this will work with memes."

Interesting, when I go to facebook (which admittedly is not very often) I only see updates and photos posted by friends in my feed.

Last July, I lead a project in support of a federal agency to analyze current business processes and identify weaknesses in the agency Section 508 office. My work focused primarily on externally accessible internet sites and one of the most common 508 violations that we encountered was the lack of ALT-text on images. This agency utilized a number of automated scanning tools and processes, but lacked any ability to efficiently remediate these errors. While we never talked more than from a conceptual standpoint, a coworker and I discussed something along the lines of what Facebook has accomplished here through the use of Google's Neural Networks. Very cool to see this advancement come to life.

This kind of systems/algorithims also allows to asign a certain semantic component to images (with a grain of salt, of course), which might enable further developments that weren't considered posible yet.

Sadly, it also brings another complete set of cases to the oh-so-anoying "but Facebook/Google/Twitter/Amazon does it!" clichés that we'll now have to deal with...

I'm waiting for the day this is just a library you pass an image to and it returns an array. No, not a SaaS. Then on my own pump.io, diaspora, redMatrix etc. it just works. My data, my images, my network. I'm not against the tech at all though. Neat

There are already pre-trained networks out there. TensorFlow comes with an example command line tool that you can pass any image and it will tell you what is in the image.

The classes that it can detect are from ImageNet, so that might be limiting.

If you were to train your own net, ImageNet is one of the biggest and most complete datasets and you'd surely use it for training. The alternative is to make your own training set, which will cost you money and/or time. For a proof of concept or initial prototype (until your business can pay for it), those classes should be enough.

ImageNet is not going to have something like "smile" in it's dataset like they showed in the video. It has all kinds of possible dog breads instead.

Maybe someone should create a website that lets volunteers label images for this purpose.

Yeah, in "Person, individual, someone, somebody, mortal, soul" there's the deeper category "smiler" which contains the categories "smirker" and "simperer". Some of the images are (note: my idea was to link to the images directly, but it isn't loading, so I had to take a screenshot and upload to imgur):

* http://i.imgur.com/Wex6pSR.png

Which I say is pretty much what you'd need to train a net to detect people smiling (amongst other things). Of course, there are some refinements you can make to improve accuracy and presentation of the results. My point was: you should begin with datasets that are readily available, and then improve on need (and if resources are available to justify the investment).

See visual genome for the next level beyond imagenet. http://visualgenome.org/

There is more information on what Facebook is doing on the research site.

https://research.facebook.com/blog/how-blind-people-interact...

Including a link to the publication that was written on the technology here.

https://research.facebook.com/publications/how-blind-people-...

I think its exciting and an honest attempt to make peoples lives better.

With all due respect to conspiracies, this is a cool feature.

Warning, that page has an auto-play video with sound.

This might be asking for too much, but why not use more of the image meta-data than these computer vision techniques?

If I were blind, I really wouldn't care that this is an image of "two people, smiling". Facebook has facial recognition, tagging, and locations. It would be much more valuable to me to say "Peter and Laura smiling at Channel Islands State Park."

They are just tagging, but there are ML solutions for describing in natural language.

Demo: http://googleresearch.blogspot.ro/2014/11/a-picture-is-worth...

I'd like to compare Facebook's image tagging with Google Cloud Vision API https://cloud.google.com/vision/ I think it would be interesting to see which one is more accurate or verbose.

I suppose it'll be like YouTube's automatic subtitles for audio. It'll do a bad, but passable, job: at least the blind and visually impaired have some idea of what the image contains.

"Cat. Cats. Cat. Baby. Dog. Baby. Cat. Baby with dog."

Bag. Duck face. Nails. Duck face.

Glad to see Mark explaining what a screen readers is to millions of people :-D

This is really what I wanted to use the Google Image API to do.

But it's way too expensive.

All I wanted was keywords for alt-text, dimensions for placeholder, and the dominant colour for placeholder background.

https://cloud.google.com/vision/

The price for that would be $7.50 per 1,000 images for the first million images.

I have some 60,000 images on the site I run and don't happen to have $450 in loose change laying around (the whole site costs less than that to run each month).

I guess I don't care about alt tags that much.

Re-upload them to facebook then scrape the generated descriptions ;)

Do you have $7.50 per month? Just do a thousand a month.

Is this tool open source? would be a great contribution to the accessibility community.

Very cool.

Obvious next step: build this into the OS/browser/screen reader.

I do wonder when somebody uses this technology to troll.

The techniques already exist[0] for some sophisticated trolling, though it may be hard to achieve in practice without direct access to the classifier being used.

[0] http://karpathy.github.io/2015/03/30/breaking-convnets/

that's honestly very creepy

Ah, the Twitter app for Android (beta version) recently added a feature that allows you to add a description to pictures you upload for impaired people.

Obligatory: https://www.youtube.com/watch?v=_wXHR-lad-Q

It's impressive that people don't really connect the dots and see that as a huge threat to their freedom.

Reply on news.ycombinator.com