Comment by Houshalter | Hacker News Reader

Comment by Houshalter | Hacker News Reader

The paper and title implies it's absorbing these stereotypes from humans. I think there is another explanation. Remember these models are trained on a dataset of news or Wikipedia articles. And it's 'goal' is to find vectors that predict what contexts words are more likely to appear in.

So if 34% of doctors are female, then you would expect 34% of doctors in news or Wikipedia articles to be female. Even if the articles are completely unbiased and the writers have no stereotypes whatsoever. And so the word vector would naturally label "doctor" something like "66% likely to occur in a male context".

And in fact this paper confirms that. Figure 1 shows that the word vectors are highly predictive of the actual gender distribution of various occupations. Probably much more accurate than most people would be. So it's not mindlessly absorbing human stereotypes. It's learning reality's stereotypes.

This result is completely expected and desirable. What makes word vectors so powerful is how they can learn complicated correlations between words and their contexts. The famous example is how it learns that "Queen" is the female equivalent of "King". Which is a gender stereotype as well. If it wasn't able to learn that doctors were a bit more likely to be male, that would be more surprising.

I think you are making a distinction without a difference. If the word vectors pick up biases from wikipedia text, than for all practical purposes, they are (indirectly) absorbing stereotypes from humans. This is an expected result, but not necessarily desirable in the end.

The parent's point is that they may not be absorbing stereotypes from humans at all. They may be generating accurate beliefs about the world from text representations of the world.

So, "plants are pleasant" or "insects are unpleasant" are universal truths?

Quite possibly. Words relating to insects will occur in news articles about malaria, zika, crop destruction, etc. Words relating to plants might occur in articles about arbor day, spring time, environmentalism, etc.

An exercise: Words relating to insects will occur in news articles about environmentalism, crop production, rituals of rebirth, etc. Words relating to plants might occur in articles about crop destruction, the international drug trade, people getting poisoned, etc.

rmxt questioned the universality of sentiment analysis. Responding by noting specific contexts, free from a clear coherent general structure, is an assertion against the discovered sentiments' universal truth.

But it is a universal truth that humans generally find plants pleasant and insects unpleasant. And the word "pleasant" is entirely based on human preferences after all.

What I'm probably missing indeed is that scoping of universality to humans. Lately I've been trying to be more explicit in my written communications in an attempt to understand both the limits of my knowledge and perceptions and the limits of the sources of information that I digest.

Is suggesting that pleasantness is a sentiment that's not unique to humans really that controversial?

super late edit: it's specifically flowers, not plants, that people are biased towards finding pleasant

"...it is a universal truth that humans generally find plants pleasant..."

Ah, but those exceptions are really unpleasant.

The example the parent poster described is "doctors are 66% male", which could very well be true.

My point is that neither "insects are unpleasant" nor "plants are pleasant" nor "doctors are 66% male" are immutable features of the universe. They are merely snapshots of the human view of world conditions, as the world is now. "True now", but not "true forever and always".

The paper seems to advocate for designing ML systems that learn that what is "true now" may not be "true forever and always". It seems to be quite the opposite of "there are certain truths that ML systems should not learn."

If your standard for truth is "immutable feature of the universe" then you might as well give up now because we don't know about any of those, or indeed if any exist at all.

Setting such a standard for a machine is ridiculous if all you want is a new tool to get some work done.

The machines are generating accurate semantic graphs of human generated text representations of human observations.

The GP is saying that the bias isn't an attribute of the Wikipedia text, but of reality.

If the reality is that only 34% of doctors are female, why is it not desirable for the machine to learn that?

Again, from humans and from reality is not different. Whether it is desirable to avoid stereotypes depends on your values.

What humans think they know is not actually reality - fine.

But then what's the difference between a fact and a stereotype, in your opinion?

A fact is a statement corresponding with reality, a stereotype is a belief/generalization deemed harmful/undesirable.

Not answering for the parent: a fact is any instantaneous snapshot of reality. A stereotype is misapplying properties of specific and limited context to a universal scope.

It depends on what you want the machine to do. If you are making a gambling machine that looks at pairs of names and makes bets as to which name belongs to a doctor, you want it to learn that.

If the machine looks at names and decides who to award a "become a doctor" scholarship to, based on who it thinks is most likely to succeed, you don't want it to learn that.

I agree that if your goal is to build a machine that decides who gets to become a doctor, you need to do more than just let it loose on a bunch of text.

But I don't think preventing it from learning the current state of the world is a good strategy. Adding a separate "morality system" seems like a more robust solution.

What do you think of Bolukbasi's approach that's mentioned in the article? In short, you let a system learn the "current state of the world" (as reflected by your corpus), then put it through an algebraic transformation that subtracts known biases.

Do you consider that algebraic transformation enough of a "morality system"?

I hope you're not saying we shouldn't work on this problem until we have AGI that has an actual representation of "morality", because that would be a setback of decades at least.

> put it through an algebraic transformation that subtracts known biases

> Do you consider that algebraic transformation enough of a "morality system"?

I would consider it a sort of morality, yes. But keep in mind that the list of "known biases" would itself be biased toward a particular goal, be it political correctness or something else.

Yes, every step of machine learning has potential bias, we know that, that's what this whole discussion is about. Nobody would responsibly claim that they have solved bias. But they should be able to do something about it without their progress being denied by facile moral relativism.

If we can't agree that one can improve a system that automatically thinks "terrorist" when it sees the word "Arab" by making it not do that, we don't have much to talk about.

A black box neural network attempting to draw inferences from a human-biased dataset - potentially even more biased because it can't understand subtexts - and then verifying that conclusion through an ad-hoc set of "morality checks" entirely independent from how it reached the conclusion sounds like a recipe for disaster.

That's even before the marketing people get involved and start claiming the system is free from human biases...

I'm not sure what your objection is with regards to the independence aspect. Why would having the morality checks integrated into the "learning about the world" part be better?

If you had an unwavering moral code which dictated that men and women should be treated equally, for example, why would it matter which facts are presented to you, in what order, or how you process them? Your morality would always prevent you from making a prejudiced choice, in that regard.

Frankly, I'm not sure the "men and women should be treated equally" instruction is even possible if the data isn't processed in a way which specifically controls for the effects of gender (some of which may not be discernible from the raw inputs).

Sure, it's theoretically possible that an algorithm parsing text about medics' credentials that (e.g) positively weights male names and references to all-boys' schools and negatively weights female names and references to Girl Guides will be on average fair after an ad hoc re-ranking of all its candidates to take into account the instruction to treat male and female candidates equally. It's just unlikely to achieve this without completely reorganizing its underlying predictive model

[1]there's an interesting parallel to ongoing human arguments about how a machine should follow its "morality checks" should do this: does it ensure the subjects are "treated equally" in terms of achieving 50/50 gender ratio irrespective of the candidate pool (thus potentially skewing it massively in favour of the side with the weaker applicants), does it try to weight results so gender balance reflects historic norms (thus permanently entrenching the minority)? Or does it try to be "gender blind" by testing all its inputs for whether they're gender biased and normalising for or discarding those which are, which is basically learning everything again from scratch...

Even if a categorization is true in a trivial sense, what generally isn't reported and thus readily inferred from fairly naive text-parsing algorithms is significant. People generally don't bother stating a perpetrator (or indeed a victim or possible witness law enforcement hopes to contact) is $majorityrace in most countries' crime reports, for example.

The machine can learn that someone named "Jamal" is more likely to be associated with the word "perpetrator", particularly in corpora centered on American news.

This is likely a true fact about the world: one that results from racial profiling and unequal enforcement.

It's not desirable to learn that, because encoding this in an AI system's belief about the "meaning" of the name "Jamal" will lead to more racial profiling.

Just because something could be considered "true" doesn't mean it's good to design systems that will perpetuate it being true.

Suppose that the system "learned" that marriage consisted of one man and one (or very rarely more) woman. Would that be "reality"? (In fact, I'd rather bet that it did, and I congratulate the authors on the wisdom of not advertising that fact.)

Various behavioral accidents can easily become embedded in culture, laws, and, yes, programs, at which point it stops mattering if they represent reality or "reality"; the real world will happily follow the cultural construction.

The distinction is very important. If it's just regurgitating human biases that would be bad. Humans often have very inaccurate and warped beliefs after all. If it's accurately modelling reality, then what's the problem? That's what we want it to do. Why would you want a less accurate model of reality?

I've seen interpretations of this result that think it's proof "language is sexist" or whatever. But there's no evidence that the humans who wrote the corups had any bias at all. As long as there are more news articles about female nurses than male nurses, the model will learn a correlation between the concepts.

Why would we want to reproduce existing structures of oppression in mechanical form? Have you noticed how automation often vastly amplifies things? It's a short step from saying 'this model accurately reflects the bias in society' to 'that's how things are, the computer says women aren't cut out to be doctors.' Surely you are aware that in real world world people rationalize decisions they don't actually understand all the time because they are not capable of or interested in improving upon the system within which they pursue their own economic interest on behalf of others whose interests do not seem coincident with their own.

> Why would we want to reproduce existing structures of oppression in mechanical form?

If (for example) 66% of Doctors are male and 34% female then it's not reproducing "existing structures of oppression" it's inferring something about reality.

If it’s saying that and only that then that’s obviously fine. However, if that knowledge is then applied in any other way, then that’s problematic.

In an environment in which Blue people are banned from becoming doctors, its also inferring something about reality to conclude that 0% of Doctors are Blue. It would be entirely wrong, however, to use these inputs to infer anything whatsoever about the respective propensity of Blue and Green people to become doctors in an environment in which such a rule or idea of a rule had never existed. Obviously "structures of oppression" - real and imagined - which lead to fewer female doctors even in western liberal democracies where women wishing to become doctors are generally met with encouragement are less extreme, but that isn't to say they don't exist or that a computer output (or human interpretation of said computer output) is likely to draw correct inferences from it.

And if you think that people won't use the idea that the outputs are unbiased because the computer isn't programmed with the same prejudices that produce the inputs, I have some algorithmically-generated investment advice involving a bridge to sell you

> It would be entirely wrong, however, to use these inputs to infer anything whatsoever about the respective propensity of Blue and Green people to become doctors in an environment in which such a rule or idea of a rule had never existed.

That's fine but it isn't the goal of these algorithms. It isn't the reality that is useful for them to learn. It's a different problem to try to build some kind of "unbiased" ontology rather than just to learn about words. Feel free to research or create solutions to this other problem, it sounds interesting.

It's inferring something about reality, but what?

Suppose, for example, that I gave this same statistic to someone and then asked them to select from a pool of 100 applicants for 50 available places in medical school. Let's assume that there's an equal # of male and female applicants and that their exam results are all similar. Do you think that knowing about this 66-34 split might influence the gender balance of the final selection?

Knowing about the gender balance wouldn't influence the final selection if you programmed the selection criteria not to be influenced by the gender balance.

The whole point of training and using machines is to make more accurate, more useful decisions in a complex world.

That can't happen if we give them data that isn't borne out by reality, or tell them to ignore data that is.

If you have to change the terms of the question to give an answer, then I think I've made my point.

>structures of oppression

What oppression? How are word vectors oppressing anyone? What a ridiculous claim.

>Have you noticed how automation often vastly amplifies things?

No, not at all. I've heard this claim on similar discussions. But I've yet to see a convincing example. Particularly with word2vec. I find it very implausible that word vectors will somehow discriminate against female doctors or whatever.

>It's a short step from saying 'this model accurately reflects the bias in society' to 'that's how things are, the computer says women aren't cut out to be doctors.'

No it's not a short step at all. No one is ever going to use word vectors to figure out what genders are capable of what jobs. At worst, your auto-correct might be slightly less likely to suggest "doctor" for a misspelled word occurring in a female context. And on net it will still make more accurate corrections than the alternative.

>No one is ever going to use word vectors to figure out what genders are capable of what jobs.

How can you possibly make this claim?

Biased word embeddings have the potential to bias inference in downstream systems (whether it's another layer in a deep neural network or some other ML model).

It is not clear how to disentangle (probably) undesirable biases like these from distributed representations like word vectors.

No one is ever going to use word vectors to figure out what genders are capable of what jobs.

Directly, no. Nobody is going to go 'ah, word2vec - a new tool with which to perpetuate patriarchal capitalism, mwuhahaha'...probably. People are weird that way.

But indirectly they certainly will. How about NPC character generators in MMORPGs? Or chatbots on social networks? Stock characters in auto-generated romance novels? The possibilities are endless.

No doubt you will these examples are ridiculous, because you seem like a rigorous scientifically minded person who would be careful not to use data in inappropriate contexts, and who would try to discount cultural or emotional factors in making strategic decisions. But you are only as good at this as your own self-awareness and willingness to acknowledge the existence of implicit bias.

And many people are quite different from you and more easily or willingly allow their judgment to be shaped by representational stereotypes. Marketing people aim to confirm their audience's worldview very closely so that consumers will be willing to identify with the commercial prompt when it arrives. Politicians and yellow journalists routinely abuse statistics to grab people's attention. And so on.

I urge you to think more about this, and in more imaginative fashion. People are often surprised by the unexpected applications of technology employed by others.

Because you cannot change reality if you do not first acknowledge what it is. First off, this is an analysis tool. If we warp our analysis tools to pretend that e.g. no gender biases exist in places where they do, then we are not making the world a better place, we are just removing our ability to quantify the ways in which it is not.

Reply on news.ycombinator.com