Comment by MereInterest | Hacker News Reader

Comment by MereInterest | Hacker News Reader

Parent Full thread

To add to the other answers, machine learning based on biased datasets results in biased models. For example, Word2Vec can be used to make analogies. Having it determine "Man is to woman as king is to ___." results in "queen", which is reasonable. On the other hand, determining "Man is to computer programmer as woman is to ___." results in "homemaker". The algorithm wasn't deliberately designed to have sexist bias in it, but there was implicit bias in the dataset from which it learned.

https://www.technologyreview.com/2016/07/27/158634/how-vecto...

As another example, predictive policing tries to place police in places with higher crime rates. Those crime rates are determined by looking at past history of police reports and arrests. That past history has human bias already in it, with disproportionately higher arrest rates in places with racial minorities. The effect of the predictive policing is to justify overpolicing of minorities.

https://en.wikipedia.org/wiki/Predictive_policing#Criticisms

The analogy thing didn’t entirely hold up: the original demonstration was constrained not to return the same word in the prompt, so “man is to woman as doctor is to ____” had to return something that’s close to, but not the same as “doctor” in the embedding space. Hence, it returns nurse.

Ironically, this makes the original point nearly as well: we need to evaluate the hell out of machine learning systems to make sure that they’re doing what we think they are and that they’re not keying off something else instead, especially something biased. To date, the field has been...not great about this.

I wonder what it would return for "Woman is to doctor as man is to ___". I'd bet money on the result not being "nurse".

It highly depends on the corpus and you can check for yourself.

I'm being lazy, but the results for Gutenberg books you can check online at http://labs.statsbiblioteket.dk/dsc/

- man is to woman as doctor is to reprovingly (nurse is the first noun, on position 4) - woman is to man as doctor is to snodgrass (after a couple nonsense/rare words)

The most important thing that teaches us is that big corpora (bigger than PG) are essential for this method.

You can monkey around with it here: http://bionlp-www.utu.fi/wv_demo/ (choose the English Google News model, but this may not be exactly the same set/model as the original report).

Man is to Woman as Doctor is to ___ gives 1) gynecologist 2) nurse 3) doctors 4) physician 5) pediatrician

Woman is to Man as Doctor is to ___ gives: 1) physician 2) doctors 3) surgeon 4) dentist 5) cardiologist

These are just generally near "Doctor" though: the ten nearest terms are physician, doctors, gynecologist, surgeon, dentist, pediatrician, pharmacist, neurologist, cardiologist, and nurse.

Some gender differences may persist (nurse is #2 for `woman`, but #68 for `man`, but it's also near `woman` generally and you could imagine it gets a bit of a boost from the verb ("to feed a baby") being attached exclusively to women too.

Anyway, my point is not that there's no bias (there certainly can be--seed GTP-3 with a prompt about Muslims) but that one should be wary of thinking they know what the model is doing.

How would you expect the computer programmer analogy be completed? I legitimately don’t know what a reasonable answer would be.

The takeaway is the model has bias. It’s up to humans to decide what to do with that information.

Depending on the input “homemaker” may be a technically reasonable output.

Should we use such a model to suggest career paths to high school students? Or should we reevaluate the methodology?

Who cares about ML when we already do that:

https://www.slideshare.net/yuyomajadero/jobs-occupations-pro...

The danger is in thinking ML can magically see past human biases and/or is not subject to it’s own types of bias.

Seems like an overhyped danger. Does anyone actually think that? You would have to not even know that the purpose of a language model is to model human language. But if that's the level of your understanding, why would you select a language model to use as a career guidance oracle? Such a negligent career advisor would be just as likely to use those children's posters with pictures of female nurses and male doctors.

I'd like to add that there's also a danger in people trying too hard to avoid bias and losing important information. For example, a man who's a homemaker instead of a programmer is a less attractive partner for a woman. So such an occupation might harm both his quality of life and that of his partner. There is some useful information encoded in cultural bias. Even if that information turns out to be entirely socially constructed, it still has real harmful effects on real humans who go against it.

> Does anyone actually think that [ML can magically see past human biases and/or is not subject to it’s own types of bias]?

I know some people who, fairly strongly, believe that ML is the future of non-biased decision making, yes.

> a man who's a homemaker instead of a programmer is a less attractive partner for a woman

That's definitely going to be [CITATION NEEDED].

> For example, a man who's a homemaker instead of a programmer is a less attractive partner for a woman.

This is much to sweeping a generalization to have a place here. You might say a man who is a homemaker is less attractive than a programmer to you, but don't speak for everybody else.

Hmmm... A Homemaker must balance a myriad of factors in order to create an environment within which a group of people can productively live and work. The Man/Woman link is the more humorous seeing as programning was originally considered women's work.

Two independent things might best be analogized as A:B as C:B. So also computer programmer.

Except that they're not independent and that looks like a fact which the model discovered.

The model should be able to tell the difference between population demographics in a particular country and a definition. Demographically, they're not independent. Definitionally, they are entirely independent.

> On the other hand, determining "Man is to computer programmer as woman is to ___." results in "homemaker"

Who are we to decide for the woman what she should feel about it? Is bias absolute or relative, objective or subjective? Is there a one true policy for dealing with bias or can there be one?

Reply on news.ycombinator.com