Semantics derived automatically from language corpora contain human-like biases
scim.ag
scim.ag
So if 34% of doctors are female, then you would expect 34% of doctors in news or Wikipedia articles to be female. Even if the articles are completely unbiased and the writers have no stereotypes whatsoever. And so the word vector would naturally label "doctor" something like "66% likely to occur in a male context".
And in fact this paper confirms that. Figure 1 shows that the word vectors are highly predictive of the actual gender distribution of various occupations. Probably much more accurate than most people would be. So it's not mindlessly absorbing human stereotypes. It's learning reality's stereotypes.
This result is completely expected and desirable. What makes word vectors so powerful is how they can learn complicated correlations between words and their contexts. The famous example is how it learns that "Queen" is the female equivalent of "King". Which is a gender stereotype as well. If it wasn't able to learn that doctors were a bit more likely to be male, that would be more surprising.
rmxt questioned the universality of sentiment analysis. Responding by noting specific contexts, free from a clear coherent general structure, is an assertion against the discovered sentiments' universal truth.
Is suggesting that pleasantness is a sentiment that's not unique to humans really that controversial?
super late edit: it's specifically flowers, not plants, that people are biased towards finding pleasant
Ah, but those exceptions are really unpleasant.
The paper seems to advocate for designing ML systems that learn that what is "true now" may not be "true forever and always". It seems to be quite the opposite of "there are certain truths that ML systems should not learn."
Setting such a standard for a machine is ridiculous if all you want is a new tool to get some work done.
If the reality is that only 34% of doctors are female, why is it not desirable for the machine to learn that?
But then what's the difference between a fact and a stereotype, in your opinion?
If the machine looks at names and decides who to award a "become a doctor" scholarship to, based on who it thinks is most likely to succeed, you don't want it to learn that.
But I don't think preventing it from learning the current state of the world is a good strategy. Adding a separate "morality system" seems like a more robust solution.
Do you consider that algebraic transformation enough of a "morality system"?
I hope you're not saying we shouldn't work on this problem until we have AGI that has an actual representation of "morality", because that would be a setback of decades at least.
> Do you consider that algebraic transformation enough of a "morality system"?
I would consider it a sort of morality, yes. But keep in mind that the list of "known biases" would itself be biased toward a particular goal, be it political correctness or something else.
If we can't agree that one can improve a system that automatically thinks "terrorist" when it sees the word "Arab" by making it not do that, we don't have much to talk about.
That's even before the marketing people get involved and start claiming the system is free from human biases...
If you had an unwavering moral code which dictated that men and women should be treated equally, for example, why would it matter which facts are presented to you, in what order, or how you process them? Your morality would always prevent you from making a prejudiced choice, in that regard.
Sure, it's theoretically possible that an algorithm parsing text about medics' credentials that (e.g) positively weights male names and references to all-boys' schools and negatively weights female names and references to Girl Guides will be on average fair after an ad hoc re-ranking of all its candidates to take into account the instruction to treat male and female candidates equally. It's just unlikely to achieve this without completely reorganizing its underlying predictive model
[1]there's an interesting parallel to ongoing human arguments about how a machine should follow its "morality checks" should do this: does it ensure the subjects are "treated equally" in terms of achieving 50/50 gender ratio irrespective of the candidate pool (thus potentially skewing it massively in favour of the side with the weaker applicants), does it try to weight results so gender balance reflects historic norms (thus permanently entrenching the minority)? Or does it try to be "gender blind" by testing all its inputs for whether they're gender biased and normalising for or discarding those which are, which is basically learning everything again from scratch...
This is likely a true fact about the world: one that results from racial profiling and unequal enforcement.
It's not desirable to learn that, because encoding this in an AI system's belief about the "meaning" of the name "Jamal" will lead to more racial profiling.
Just because something could be considered "true" doesn't mean it's good to design systems that will perpetuate it being true.
Various behavioral accidents can easily become embedded in culture, laws, and, yes, programs, at which point it stops mattering if they represent reality or "reality"; the real world will happily follow the cultural construction.
I've seen interpretations of this result that think it's proof "language is sexist" or whatever. But there's no evidence that the humans who wrote the corups had any bias at all. As long as there are more news articles about female nurses than male nurses, the model will learn a correlation between the concepts.
If (for example) 66% of Doctors are male and 34% female then it's not reproducing "existing structures of oppression" it's inferring something about reality.
And if you think that people won't use the idea that the outputs are unbiased because the computer isn't programmed with the same prejudices that produce the inputs, I have some algorithmically-generated investment advice involving a bridge to sell you
That's fine but it isn't the goal of these algorithms. It isn't the reality that is useful for them to learn. It's a different problem to try to build some kind of "unbiased" ontology rather than just to learn about words. Feel free to research or create solutions to this other problem, it sounds interesting.
Suppose, for example, that I gave this same statistic to someone and then asked them to select from a pool of 100 applicants for 50 available places in medical school. Let's assume that there's an equal # of male and female applicants and that their exam results are all similar. Do you think that knowing about this 66-34 split might influence the gender balance of the final selection?
The whole point of training and using machines is to make more accurate, more useful decisions in a complex world.
That can't happen if we give them data that isn't borne out by reality, or tell them to ignore data that is.
What oppression? How are word vectors oppressing anyone? What a ridiculous claim.
>Have you noticed how automation often vastly amplifies things?
No, not at all. I've heard this claim on similar discussions. But I've yet to see a convincing example. Particularly with word2vec. I find it very implausible that word vectors will somehow discriminate against female doctors or whatever.
>It's a short step from saying 'this model accurately reflects the bias in society' to 'that's how things are, the computer says women aren't cut out to be doctors.'
No it's not a short step at all. No one is ever going to use word vectors to figure out what genders are capable of what jobs. At worst, your auto-correct might be slightly less likely to suggest "doctor" for a misspelled word occurring in a female context. And on net it will still make more accurate corrections than the alternative.
How can you possibly make this claim?
Biased word embeddings have the potential to bias inference in downstream systems (whether it's another layer in a deep neural network or some other ML model).
It is not clear how to disentangle (probably) undesirable biases like these from distributed representations like word vectors.
Directly, no. Nobody is going to go 'ah, word2vec - a new tool with which to perpetuate patriarchal capitalism, mwuhahaha'...probably. People are weird that way.
But indirectly they certainly will. How about NPC character generators in MMORPGs? Or chatbots on social networks? Stock characters in auto-generated romance novels? The possibilities are endless.
No doubt you will these examples are ridiculous, because you seem like a rigorous scientifically minded person who would be careful not to use data in inappropriate contexts, and who would try to discount cultural or emotional factors in making strategic decisions. But you are only as good at this as your own self-awareness and willingness to acknowledge the existence of implicit bias.
And many people are quite different from you and more easily or willingly allow their judgment to be shaped by representational stereotypes. Marketing people aim to confirm their audience's worldview very closely so that consumers will be willing to identify with the commercial prompt when it arrives. Politicians and yellow journalists routinely abuse statistics to grab people's attention. And so on.
I urge you to think more about this, and in more imaginative fashion. People are often surprised by the unexpected applications of technology employed by others.
best probably to read that one first.
Unless the link was changed in the few minutes since you posted your comment, the link for the article is the original Science paper (http://science.sciencemag.org/content/356/6334/183.full)
Page with actual link:
http://science.sciencemag.org/content/356/6334/183/tab-pdf
Link to PDF itself:
http://science.sciencemag.org/content/sci/356/6334/183.full....
If I look at glove & WordNet usage e.g. for topic extraction, bagging / clustering or semantic similarity would you say we would need to get rid of such a bias, e.g. create something like a Geiger counter for NLP.
Alternative view - when doing sentiment analysis / classification would you say that such a bias actually helps to identify a type of sentiment in a doc / sentence.
- An AI correctly infers (simply by reading text) that a physicist is male and a nurse is female.
- An AI correctly infers the gender of humans with androgyonous names.
- An AI infers insects are unpleasant and flowers are pleasant to humans.
- An AI also infers that African American names are more likely to be associated with unpleasantness than European names.
[edit: to those who dislike this comment, can you tell me what you object to? Which of my concrete examples is not in the paper?]
Near as I can tell, your paper shows that these "biases" result in significantly more accurate predictions. For example, Fig 1 shows that a machine trained on human language can accurately predict the % female of many professions. Fig 2 shows the machine can accurately predict the gender of humans.
Normally I'd expect a "bias" to result in wrong predictions - but in this case (due to an unusual redefinition of "bias") the exact opposite seems to occur.
(Drawing on your analogy with stereotypes, it's probably also worth linking to a pointer on stereotype accuracy: http://emilkirkegaard.dk/en/wp-content/uploads/Jussim-et-al-... http://spsp.org/blog/stereotype-accuracy-response )
"In AI and machine learning, bias refers generally to prior information, a necessary prerequisite for intelligent action (4). Yet bias can be problematic where such information is derived from aspects of human culture known to lead to harmful behavior. Here, we will call such biases “stereotyped” and actions taken on their basis “prejudiced.”"
This definition is not unusual. This is about inferences that are wrong in the sense of prejudiced, not necessarily inaccurate.
In any case, the paper suggests that this "bias" or "prejudice" is better described as "truths I don't like". I'm asking if the author knows of any cases where they are actually not truthful. The paper does not suggest any, but maybe there are some?
Think for example of an inductive bias. If I see a couple of white swans, I may conclude that all swans are white, and we all know this is wrong. Similarly, I may conclude the sun rises everyday, and for all practical purposes this is correct. This kind of bias is neither wrong nor right, but, in the words of the article "a necessary prerequisite for intelligent action", because no induction/generalization would be possible without it.
There are undoubtedly examples where the prejudiced kind of biases lead to both truthful and untruthful predictions, but that seems beside the point, which is to design a system with the biases you want, and without the ones you don't.
From what I understand, the fear surrounding embedding human stereotypes into ML systems is that the stereotypes will get reinforced. In some way or form, there will be less equality of opportunity in the future than exists today, because machines will make decisions that humans are currently making. Societal norms evolve over time, yet code can become locked in place.
Is your takeaway from this paper that we, as the creators of intelligent machines, should allow them to continue to making "positively" right assumptions simply because that's the way we, as humans, have always done them? Is "positively" right, in your opinion, in all cases equivalent to "normatively" right?
Given how accurate human corpora is at predicting things like gender distribution in jobs for instance, wouldn't making an "unbiased" corpora make an inaccurate AI?
Shouldn't we be careful in implying things like the biases and solutions to said biases? For instance, I'd like to know if my algorithm for filtering job applicants is trying to undo the injustices of the world in addition to finding the best candidates.
> Our findings are also sure to contribute to the debate concerning the Sapir-Whorf hypothesis (17), because our work suggests that behavior can be driven by cultural history embedded in a term’s historic use. Such histories can evidently vary between languages.
No, not really. Linguistic relativity makes a claim about the direction of causation--from language to thought. This study does nothing to test the direction of causation, which is usually considered possible only with controlled experiments. This chicken and egg debate has been going on for the better part of the last century--it is not an easy problem.
> Our results also suggest a null hypothesis for explaining origins of prejudicial behavior in humans, namely, the implicit transmission of ingroup/outgroup identity information through language. That is, before providing an explicit or institutional explanation for why individuals make prejudiced decisions, one must show that it was not a simple outcome of unthinking reproduction of statistical regularities absorbed with language.
Once again, this was not an experiment capable of producing causal evidence. The authors have shown that human biases can be replicated by statistical learning of language corpora--admirable work, but nothing new here for Sapir-Whorf.
This is an extraordinarily bold claim. I'd be quite interested in how peoples' responses to the article changed if this was the lead.
Very bold. The full quote is worth reproducing:
> Our results also suggest a null hypothesis for explaining origins of prejudicial behavior in humans, namely, the implicit transmission of ingroup/outgroup identity information through language. That is, before providing an explicit or institutional explanation for why individuals make prejudiced decisions, one must show that it was not a simple outcome of unthinking reproduction of statistical regularities absorbed with language.
I'm reminded of Parable of the Polygons[0] which illustrates Shelling's model of segregation, showing how quite small initial biases can be amplified and result in very large segregation.
It would be seem very sad if tribalism in all its forms is simply an emergent behaviour, a result of a random fluctuation (e.g. one or two racist individuals) causing a chain reaction throughout society, where biases become gradually amplified even if most individuals are, themselves, generally well-meaning. How do we escape from that?
Sadly, that seems to be the case.
I wouldn't even turn racism into a "special case" here. Humans are capable of dividing themselves into ingroups and outgroups over everything, no matter how trivial. I suspect the segregation process will occur with all in/outgroup divisions. Separations along the race and gender lines are particularly prevalent because those are the most obvious, noticeable differentiators between people.
I get that you like to take contrarian positions. But you also make a habit of inserting flamebait into your posts about them. This combination is trolling. If you continue to do this we will ban you.
Specifically, we need you to stop playing the following game on HN:
1. Post contrarian view
2. Include provocation
3. People get provoked
4. Act like people can't handle your truth
We detached this comment from https://news.ycombinator.com/item?id=14116469 and marked it off-topic.
The article explicitly recommends building systems which can't learn those things, and suggests characterizing them is a first step:
"We recommend addressing this through the explicit characterization of acceptable behavior. One such approach is seen in the nascent field of fairness in machine learning, which specifies and enforces mathematical formulations of nondiscrimination in decision-making (19, 20). Another approach can be found in modular AI architectures, such as cognitive systems, in which implicit learning of statistical regularities can be compartmentalized and augmented with explicit instruction of rules of appropriate conduct (21, 22)."
According to the article, the most closely related work "is concurrent work by Bolukbasi et al. (6), who propose a method to “debias” word embeddings."
(Recall that in the context of the article, "bias" is something that generates true predictions that are objectionable rather than something which is false.)
I've made many attempts to explain, don't believe anyone could accuse us of being impatient with you, and do believe you're more than smart enough to understand. If your account consistently produces troll effects on HN, which it does, then at some point it's you who are responsible—not people who can't handle the truth, don't want 'man' to know things, don't know math, or however else you blame others. At some point enough is enough.
If you really need a further explanation I'd be happy to try, but would need some indication that you're asking in good faith.
Alternative and less biased conclusion: language corpora reflects statistical distributions in the society.
Sure, some of yummyfajtias comments are a bit trollish but he points out real contradictions in submissions and other comments. I have his threads bookmarked and visit them directly to check my thinking. It would be a huge loss for HN.
It's like the Socrates thing all over again. You would be banning someone offering interesting insight for upsetting the social order of the place.