Zero-shot learning: Using text to more accurately identify images | Hacker News Reader

Zero-shot learning: Using text to more accurately identify images

code.fb.com

117 pointspesenti5 comments

> WHY IT MATTERS:

> To become more useful, computer vision systems will need to recognize objects they have not specifically been trained on. For example, it is estimated that there are more than 10,000 living bird species, yet most computer vision data sets of birds have only a couple hundred categories.

I think it matters more for Facebook that they can apply these techniques to learn more about the 4 billion people that are not on Facebook, I don't think they care that much about birds ;)

However, where do you find a nice and detailed text description of you?

next on facebook: block accounts randomly and reenable them if you can describe a friend

They have been doing something like that for years: if you get locked out of your account, they show you pictures of friends and ask you to identify them. I know this has nothing to do with what the article describes, I'm just stating this for the record :P

https://www.androidtipster.com/wp-content/uploads/2016/08/fa...

(It's an old Mac and an old design of Facebook... Just so you can see how old this screenshot is!)

other social media platforms, news articles, all viral media that get reshared on facebook, all the pictures of you your mom uploads to Facebook with comments from your grandma about how cute you are, etc.

EDIT: I'm not saying Facebook will use this specific AI technique to spy on you if you are not on Facebook. They will use to understand other things than birds (like humans - ones on facebook, and ones not on facebook).

I'm sorry, but I work in this field and I find this so amazing that I don't have any sensible comments to make.

I still find the fast.ai devise paper demo[1] amazing, and that is just working on labelled but unseen classes via the ImageNet hierarchy.

In this case they do really fine grained classification on unseen classes, with no hierarchy and instead exploit plain (noisy!) text.

This visual pivot thing is something I'm going to spend the next year coming back to I think.

[1] https://github.com/fastai/fastai/blob/master/courses/dl2/dev...

Reply on news.ycombinator.com