Racial-profiling software shows how AI can be dangerous | Hacker News Reader

Racial-profiling software shows how AI can be dangerous

qz.com

42 pointsknown16 comments

It's not about intent at all, frequently in cases along the lines of "AI did racial profiling" the effect was entirely unintentional, it was the expression of a bias in the data, or an unintended bias from the point of view of the developers.

Yes, but when you seek your ML product as the panacea to all problems, you should be accountable for what it outputs but people don't want that; they want the ML to be viewed as something beyond human understanding, something omnipotent and all-seeing.

Which opens the door for people are all too willing to exploit that through real ML algo & data weaknesses.

People can be, and often are, racist without being intentionally racist.

Thank you for this. This fact needed to be pointed out.

(Edit: This wasn't sarcasm... Not sure why all the downvotes)

Not one of the downvoters, but the general mindset is that if you thank someone and emphasize their comment, you should've just upvoted because you didn't really add anything to the thread.

Exactly. I wrote a relevant comment[1] on a different thread a while ago, which I'll reproduce here for convenience:

The other day out of curiosity I was searching for basic words on ImageNet: cat, dog, man, woman.

Here is the page for "woman" http://image-net.org/search?q=woman

I was surprised to notice a couple of things:

- There are categorizations for "mistress", "slut", and "adultress". The closest analogs for "man" appear to be "playboy", "stud", and "pimp". There is also a "foolish woman" and doesn't appear to be an analog for man.

- Look at the pictures for "mistress". Many if not most of them are just women in their bathing suits or doing other normal things, like laying in a bed with a man. The same is true for "slut". Does this mean AIs using ImageNet could look at a woman on a beach and categorize her as a "slut"?

- There is a category for "honest woman", which appears to be three pictures of a woman standing next to a man.

I'm not saying anybody is at fault or that the people who put ImageNet together did this intentionally. After all, it does reflect the (unfortunate) reality of how our society often perceives images of women.

But before I poked through that dataset, it wouldn't have occurred to me how important it is that we focus on equality and non-discrimination when it comes to AI. This declaration[2] seems like an interesting step.

[1] https://news.ycombinator.com/item?id=17103921

[2] https://www.accessnow.org/the-toronto-declaration-protecting...

I don't know how much stock I would put in this being indicative of our society's views on gender. Notably, you point out that there is no male analog for "foolish woman", but we're all familiar with the foolish man trope (Joey from friends, Phil from Modern Family, Tim from Home Improvement, Ron Burgundy from Anchror Man, etc etc).

Note: I shouldn't have to point out that I'm _not_ saying our society is fair and just with respect to gender, but I know a lot of people will try and twist my words, so hopefully this disclaimer heads off a lot of unnecessary tangents.

It makes me think less "failure in image classification", and more "implicit bias in the WordNet database and/or failure to represent concepts that can't be described succintly in English".

It could also be "failure to pick words that mean the thing the image classifier learned". Also, implicit bias in the WordNet database could be _a different implicit bias than exists in society_. Simply, we don't know and we should avoid pat explanations and guard ourselves against our own confirmation biases.

Usually its "idiot guy" not "foolish guy"

Don't forget those trollopy Flibbertigibbets

The slut categorization is Correct. There is no such analog as a slut for males. Females choose the mating structure not men. A man is not able to sleep with unlimited women, yet most women can sleep with unlimited men if they desire.

Mistress has multiple definitions and is more a social construct than something you can see in a picture so I'm not sure how they come to a conclusion on that one. There do seem to be lots of BSDM style photos in that set which makes sense as that is the only discreet physical definition of a "mistress"

"foolish woman" Usually people say "idiot guy" but that comes up with Television which is often referred to as "Idiot box"

"Honest Woman" is defined on their site and seems to search accordingly. Also it is a common saying which probably matches the expected results of most searchers. Synset: honest woman Definition: a wife who has married a man with whom she has been living for some time (especially if she is pregnant at the time); "he made an honest woman of her".

The classification seems to be based off common speech usages and seems to be adhering to them.

> There is also a "foolish woman" and doesn't appear to be an analog for man.

There is Incompetent, Weakling, doormat, wuss all goes to images of men.

ImageNet reflect the unfortunate reality of how our society perceives gender roles. You ask it to produce a image of an antagonist, you get a male picture. A lover and you get a picture of a woman. A foolish woman and you get a woman, a incompetent man and you get a man. A good person and you get a mix of women and men. A bad person and its all men.

If we had a person look at the category titles and just guess based on gender roles what the gender would be, I think we would see a rate of 90% or higher in correct guesses. At this point we have not yet introduced the AI or any potential bias that AI can have.

> “Ultimately, it’s about intent and we have a lot of faith in humanity,” [Zeiler] wrote. “It’s not our business to limit what developers create and we choose to believe that most people will use our Demographics for good"

It would be less insulting to the reader's intelligence if he outright said that racial profiling is just too lucrative of a market to pass up.

> Privacy advocates like the American Civil Liberties Union already decry the use of facial-recognition AI in most cases, making the case that widespread adoption of the technology would mean we would live under constant surveillance by police or large tech companies. Ethnicity recognition is a step even further past the ACLU’s line in the sand.

Reply on news.ycombinator.com