Surprised they weren't more careful with this after Google had the same problem a few years ago.
https://www.theverge.com/2018/1/12/16882408/google-racist-go...
https://www.theverge.com/2018/1/12/16882408/google-racist-go...
In the Google photo case, I've seen the picture in question, and it's not difficult an all to see how a rudimentary statistical algorithm would mistake the photo for a gorilla. No, no human would make this mistake, but due to the lighting in the photo and the woman's hair it's really not hard to see how that mistake would be made, similar to how Tesla's AI can mistake the sun, low on the horizon, as a yellow traffic light.
The algorithm was not "racist", it just didn't have the social context that describing black people as apes and monkeys has a long racist history. And that is really the fundamental problem with most AI these days - it can get very good at statistical inference, but it doesn't have the background knowledge and logic to be able to "think" in the same way humans do.
In a similar vein, I recall someone lamenting a couple years ago how Google's street directions would never mispronounce "Malcolm X Boulevard" as "Malcolm 10 Boulevard" if they had any black programmers. Again, given 99% of the time when you see "X" in an address you'd pronounce it as 10, it's not hard to see how this could happen. The problem is that some errors are much more offensive than others, and AI can't really reason about that.
Yes, humans are primates. But this wasn't a discussion about taxonomy in biology class. There are lots of cases where primates is used in everyday language where it is clearly intended to refer solely to monkeys and apes. When I go to the "primate house" at the zoo I'm not going to expect to see naked humans behind the glass.
This algorithm is clearly broken, it doesn’t label all people as primates, it labels Black people wearing a green T-shirt possibly as such, I mean I’m sorry but that algorithm is definitely broken. The person in question should not be labeled as primate by any means not should any other humans unless the context is specifically some form of species taxonomy.
Also your reaction seems to be all daisies and roses, and I don’t blame you for your social context ignorance you displat, I guess you are just an engineer after all..
Also which teams at apple made the algorithm? Since you claim to know they were all 'white'.
In reality, I'm sure everyone working on this team was well aware of these potentials for training set bias before most of us even knew what ML was.
Even if this was considered it'd be pretty risque to have an employee dedicate time specifically to differentiating black ppl from apes in the algorithm.
And to your second point about how would we know if the team was “all white,” of course we don’t know that and it’s probably not true. What we do know is that tech in the us is disproportionately whiter than the general public, and ~10 person teams with no black people on them are the norm, not the exception.
Edit: I should say that 10 person teams without black people on them are the norm in FAANG companies and others headquartered in the Bay Area. Other tech hubs in the US are much more diverse.
For all you know the entire team that worked on this project could've been black. Or zero of them could've been white etc.
We can and should have a conversation about training sets and ML ethics. Resorting to unprovoked racist attacks is quite a counterproductive approach imo.
Words are fine, but person names or abbreviations will often use the English pronunciation which can be inexact or even incomprehensible.
For example: In Brazil, the major roadways are named with the initial code of the state, or BR if it's a federal road, and a bunch of numbers. So BR-101, for example. These are never named in full, so someone reading "BR-101" would not say "Brasil 101", they would say "B R 101".
There is a state called Santa Catarina, with initials SC, but Google decides that we somehow get transported to the United States somehow, since it reads "SC-406" as "South Carolina 406".
A human would at the very least have the context that SC definitely is not a common abbreviation for South Carolina in Brazil, even if they did not know any state names.
It has gotten better, but even in 2018 it should have been in- excusable to be mispronouncing names that badly.
One of the big issues with relying on AI is training data, and the responsibility for an inadequately trained AI model falls on the people who judged it ready for use, but that doesn't mean the outcome of the AI isn't racist. An AI doesn't need to know the sociopolitical history of race to be doing racist things.
Also, I've never heard of a location with Roman numerals in the name. I agree that programming in the context of knowing who Malcolm X is into a map program might be asking a bit much, but I think an "easier" implementation (reading them as letters) would have also done the right thing.
At least in Europe, extremely common. In Paris you have Boulevard Henry IV, in Lyon the street named after the same king. There's thousands of kings and queens and many more roads named after them. There's also schools, buildings and such named after them.
Ways to actually fix the problem:
1. Probe the decision boundary between these two classes in your training and test sets. I.E. look for the humans closest to being misclassified as primates. You could probably quickly get a sense if you are near/at risk of making this misclassification. You may also possibly find 1 or 2 mislabeled examples that are throwing things off and can be corrected.
2. Boost your training set by labeling more unlabeled images that are near this decision boundary.
So my point was just: this was an issue that could have been anticipated because it has already happened before. It is unfortunate that this issue came up when there are things you can do (such as the steps above), which I believe could have totally squashed this issue with enough iteration.
I can't know for sure that this is not what the Facebook team did. If they did take care to try and avoid this harmful model confusion, then I would be very surprised given my experience with deep learning and computer vision.
At a minimum, it would have been pretty trivial and a low impact to users to remove the primate class if they didn't have the time/bandwidth to really investigate this more thoroughly as I've described.
You can try to argue "hey no one should get upset about this confusion because computer vision is hard and mistakes happen", but I don't think that's a very solid argument either given the long and relatively recent history of racists misclassifying a particular subset of humans as primates.
They were careful, otherwise they wouldn't have labeled it "primates", but e.g. "monkeys" instead.