Hacker News Reader

In modern linguistics grammar is used descriptively rather than prescriptively, i.e. it's about describing how speakers actually use language rather than being about telling them how to use language 'properly'.

If there's gender bias in certain words it might be interesting to point that out but it's not the linguists' job to 'de-bias' the grammar and the underlying model.

There's an undeniable gender bias in certain words (men probably are less frequently referred to as 'beautiful' than women while on the other hand 'genius' likely is more often used when referring to men). Glossing over that by smoothing models not only misrepresents how speakers use language but probably doesn't really help the cause either.

If the corpus used displays a disproportionate association of certain words with one gender this could just mean that the corpus is insufficient for representing a language in general (as opposed to just a particular register or sociolect), which is a common problem in computational linguistics, not just when it comes to gender biases.

> it's not the linguists' job to 'de-bias' the grammar and the underlying model.

Sure, but this is not a project of linguistic research. It’s a tool that could see real-world usage. Not perpetuating the stereotypes that suffuse the data we feed into such models does seem like a worthy Endeavour, and I find it encouraging to see the creator taking these concerns seriously in this threat.

(As opposed to the community-at-large, which quickly send them to the bottom of the threat)

The issue is that the undesired language constructs one would like to remove are not universal. What is an improvement for one user group will be a regression for another & vice versa.

I’m a white, able-bodied, mostly heterosexual man, and I reject the theory that (non)discrimination is a zero-sum game.

Debiasing word vectors is not guaranteed to yield less discriminatory results in every application. In many languages, gender is expressed grammatically and there are different words for e.g. male and female scientists. If you debias the word vectors to remove associations with gender, it becomes impossible to distinguish those two. Now if you translate between two languages with that distinction, the model will have to fix itself to one of the two possibilities, and the translation error will be minimized by always picking the more common form. "First year of all-female Nobel prize winners and machine translation gets it totally wrong!" By debiasing the word vectors, the model has become more biased and less accurate overall.

If you create a discriminatory machine learning model, it's unlikely to be because you didn't debias your input representation. More often, you're training it on a task whose real-world statistics are biased, and the model learns to accurately reflect that bias to solve your task. The solution to that is not to modify the input, but rather the output you expect the model to provide.

Ah yes. This approach to ML bias is popular in some large corporate research groups:

- Say a lot of nice things about fairness

- Talk about the importance of de-biasing and doing it as well as possible

- Show (legitimately) that the farther downstream in your ML process you apply de-biasing, the more sound the results are

- Assume that any successful ML model will be used downstream in something else

- Therefore, never de-bias anything, except the final output of something where your company benefits from showing fairness: that is, a demonstration at a conference talk about fairness

- The time to de-bias real applications is always "later" and everyone can say they are working on doing the right thing

People who release data artifacts have the ability to de-bias now instead of later. It's not perfect, but it is good. If you rely on the developer after you to do the de-biasing that you're not doing, you're ensuring that de-biasing won't happen, because they won't do it either.

My point is that the appropriate debiasing is task-specific. You can completely eradicate bias one one task, but as soon as someone builds on top of your results, the bias is going to creep right back in, and likely to be worse due to the information loss in previous layers. If you just hand a bias-conscious developer a bunch of debiased word vectors with the implication that it will make their model less biased, I don't think that's really helpful. Like security, bias in machine learning needs to be addressed on the level of complete systems, it's not something you can leave to some simple input sanitizer.

That's because what's called 'debiasing' is not actually removing bias but adding another opposite one at the superficial layers we can actually notice.

That's possibly a fair description of the kind of as-late-as-possible de-biasing that I find insufficient.

I think there is not nearly enough research into de-biasing at training time, perhaps because it would be discouraged by the reward structure of our field.

("You mean I can make my model more complex and my paper longer, and in return I lose the 0.15% accuracy gain that's the entire reason I could call the result 'state of the art'? Sure, I'll get right on that!")

Ah. In that case, I agree with where you're coming from, but not on the details:

I think there's no reason to believe that bias from multiple layers of ML end up neatly rolled up into one wad of bias that can be extracted at the end of the process. If there is an empirical demonstration that this can happen, I'd love to see it.

And I think the cost of de-biasing is much lower than you think it is. I mean, I de-bias word vectors. The accuracy loss on intrinsic evaluations from doing so is tiny; it's much smaller than what you gain from easy wins that not everyone does, like improving your OOV lookup strategy (another way to improve ML results by intervening manually on your insufficent data!).

A simple example for a thought experiment: let's say you're classifying movie reviews for sentiment, so you've got a word-embedding layer (trained on general word embeddings) and a classifier layer (trained on the specific data).

The word-embedding layer will learn from corpora that it is negative to say "gay". The classifier layer will learn from the specific dataset of movie reviews that it's negative to say "Steven Seagal" (this actually happens in simple classifiers trained on the MovieLens data set). And sure, I could accept this one as objective truth, it's just an amusing and memorable example.

I think that if you save all the de-biasing for the end, you are likely to be able to fix the "Seagal" problem but not the "gay" problem, whose representation has become too complicated by going through another step of machine learning.

So I don't see the moral hazard of doing more de-biasing, as long as nobody presents it as a one-stop fix to the problem, which I sure don't.

There’s two purposes for using such a corpus, one is to get an accurate representation of how language is used. Another is to use it as part of a machine human interface, in which case you want to be careful about whether you offend your customers or users or whether you’re even violating the law by biasing certain decisions somehow.

The corpus alone can't be sufficient. The corpus displays associations that are disproportionate for how we want to use it, because corpora are from the past. But the point of ML is to build something that will be used in the future. [1]

There is no objective way to describe this difference between your training set and your test set, because you don't have data from the future. It requires a conscious decision. Enforcing that the future should be like the past is a conscious decision, and it's a lazy and unfortunate one.

This is just one instance of a problem that ML is already quite familiar with, and has been since "ML" was called "statistics": observed data produces a biased estimator of the actual distribution. You always have to correct your objectively-measured distributions for what you can't measure objectively. Here I mean "biased" in the mathematical sense; when it comes to human biases encoded in word embeddings, it also produces bias in the ethical sense. But there are many simpler instances of this.

For example: a maximum-likelihood language model assigns a probability of 0 to any word it has never seen. Maximum likelihood is, when measured, a better model of the input data than anything else. If you implement this completely objective language model, it would be useless when applied; it would output the impossible probability of 0 for most inputs. Instead you have to "smooth" and "correct" it for the fact that other words exist.

[1] https://twitter.com/random_walker/status/975700725807439879

Personally I am rather uncomfortable with the idea of 'correcting' the learning process to have the machine output what we think it should instead of what it objectively learns from the dataset. It is trading a bias against an ideal world with a bias against reality.

Where did you get the idea that data, of all things, is objective? Data is shit! Data is the noxious raw material that we have to process with great difficulty into something useful!

If you are familiar at all with machine learning, you should recognize that human decisions affect every step of the process, especially the part where the data is produced and collected. It is not an oracle of objective truth.

And let me quote Arvind Narayanan for why you are not going to get the right answer in your search for objectivity: "Training data is from the past and test data is from the future. We use ML because we want to learn from the past, not reproduce it." [1]

[1] https://twitter.com/random_walker/status/975700725807439879

IIRC, if you remove the 'uh' and 'hmm' from audio samples to train for speech recognition, you get lesser performance than if you leave them in. So it is not that easy to differentiate a-priori between actual data and noise. If you look at the history of NLP we went from 'sentences are instances of a well defined grammar' to 'sentences are a bunch of statistically related tokens'. Embracing the mess is how we got better results !

And as for de-biasing gender terms, It makes the assumption that the gender differences in language only have nefarious purposes, but without an actual proof that this is the case, you may very well throw the baby with the bathwater.

A good example of this is when the french government mandated that Resumes must be anonymous and not mention sex, gender, etc. It actually resulted in worse outcome for people from lower socio-economic background. It turned out that on average people were more forgiving of bad Resumes if it came from people who could be expected to be disadvantaged.

You've given an example of where a human decision in a data-processing pipeline harms the result. There are many examples where human decisions are a benefit. Look at every other decision in a successful speech-recognition pipeline for examples. We're not just putting raw waveforms into a black box handed to us by aliens.

If you read the paper, you'll see that de-biasing is not based on assumptions, it is also based on data. Bolukbasi ran a pretty substantial crowdsourced survey to find the comparisons that word vectors make that people consider inappropriate. Don't make hollow demands for proof when you're not even aware of what's already been shown.

Crowdsourcing comes with its own set of biases, of course, and we may want to revisit this data sometime, but so far this is a pretty reasonable proxy for whether an ML model will cause problems when it is deployed. And it's much better than nothing, the option that I'm struggling to understand why you prefer.

I read the Bolukbasi paper and I now see where you are getting at. I think that the problem is not one of bias but rather that words vector embed the whole meaning regardless of the context. And that debiasing is rather stop-gap solution as there is an endless list of bias to correct depending on the usage of the model.

You may have fixed PROGRAMMER - MAN + WOMAN = HOMEMAKER, and still get SOLDIER - AMERICAN + ARAB = TERRORIST, the list is endless

Yep, so there's a lot of work to do. Bolukbasi, Cheng, et al. will straightforwardly admit that their model only applies to the one axis of gender bias. Fortunately there are other people working on this too, and I am one of them.

The baked-in assumption that Arabs or Muslims are terrorists, or that terrorists are Arabs or Muslims, is something that the de-biasing process in ConceptNet Numberbatch (which I make) attempts to mitigate at the same time as gender and racial bias. And of course there is much more to do.

It's a fascinating and productive field of research. Why did you have such a negative initial reaction to it?

Because I find in AI's candid responses to the questions we ask a fascinating opportunity to understand humanity better. When we try to conform those answers to our desired biases we learn less from it. But now I understand that such aspects are a subset of the greater problem of getting our models to answer in a specific context.

Ultimately the Model would need to detect & learn the contexts by itself.

And more to the point; I see you are publishing data sets after correcting them for fairness. I think the unfair results can sometimes be more useful. I once prototyped a short-story tagging & recommendation system based on word2vec and I would not be surprised if raw vectors gave better results. People's taste in litterature (especially romantic) are very dependant on gender stereotypes.

Interesting, definitely need to think about debiasing. Seems like it won't really work straight out of the box since it'd destroy the 1 bit-ness of the vectors.

Though if only a few of the vectors are de-biased then you can still save a lot of space since all the other vectors are still represented using 2 numbers (while the de-biased vectors are represented using the full range of 32 bit numbers).

There may be a way to build it into the loss function so that it happens before the quantization, right?

(and holy crap, look how fast the HN conservatives are getting to my comment)

Hm not sure, would need to think more about this -- definitely an interesting idea though!

Thanks for being willing to discuss it. And sorry if I'm complicating your task.

But any interesting release of NLP data has the potential to affect the way the field progresses, so take it as a compliment that I consider this an interesting release of NLP data. That's why I'm asking you to actively consider the downstream effects of word vectors and find out if you can make them better.

Thanks for the compliments! And thanks also for bringing up the debiasing! It's interesting to see what people care about and any conversation that leads to new ideas is a plus.

Literally everything is either Left or Right. Anyone who disagrees with you is definitely a sexist conservative. No other possibility. The center is a black hole.

One of the closest related words is lady, and then effeminate, so it's quite unclear exactly what there is to be upset about? In fact it appears it's saying men are related to being effeminate, which should get you excited.

What a great quantitative evaluation of bias you just carried out.

Alright let me try to be real. After reading http://www.sciencemag.org/news/2017/04/even-artificial-intel... and https://en.wikipedia.org/wiki/Implicit-association_test (in that order thank you), it looks like the implicit association test is a good way to magnify small differences (I'm thinking floating point error) in how much you associate one of a few words with a certain other. So, presumably, quantification would really be a natural way to eliminate those types of subtle differences.

I'm glad we could at least get you to read it, but you're still not caught up. The effect is many orders of magnitude larger than floating point error.

The notion that implicit bias could be corrected by quantization (not "quantification") is an interesting hypothesis, with a low prior probability, which could be tested by experiment and easily published if it is true.