Coauthor here. The blog post is written in relatively non-technical language for a general audience, but our paper has tons of technical details that HN readers might enjoy. Give it a read!
The results are not too surprising, as the models for learning word embeddings like GloVe, word2vec, etc. learn to map to vectors existing relationships between words in training corpora. If a corpus is biased, the embeddings learned from it will necessarily be biased too.
However, the implications of this finding are wide-ranging. For starters, any machine learning system that relies on word embeddings learned from biased corpora to make predictions (or to make decisions!) will necessarily be biased in favor of certain groups of people and against others.
Moreover, it's not obvious to me how one would go about obtaining "unbiased" corpora without somehow relying on subjective societal values that are different everywhere and continually evolving. You have raised an important, non-trivial problem.
I don't believe that problem will ever be completely solvable. But I think the road to go is to make these assumptions always explicit. I.e. when the machine learning system derives a result, program it to additionally return a proof of how it came to this result. And also give a way to let the ML system return a list of all axioms and derivation rules that it has currently learned, so that they can independently be checked how much they are biased and can thus be corrected.
LIME[1] is a nice start, though.
[1] https://www.oreilly.com/learning/introduction-to-local-inter...
http://cs231n.github.io/understanding-cnn/
> Google is currently working systems that use a trillion features - I can't imagine returning some kind of rule list for that.
As I wrote: It would already help if the ML system as a first step returned the derivation with only the rules that were concretely used for a concrete derivation - this list is much shorter and can thus much easier be checked.
This is not true.
Here's an oversimplified example. Suppose your machine learning system wants to predict something, e.g. loan repayment probabilities. One input might be a written evaluation by a loan officer.
When trained on a corpora of group X, the predicted probability might be:
pred = a*written_evaluation + other_factors
However, now lets suppose the written evaluation is biased to the tune of 25% against group Y. I.e., group Y has written scores that are 25% less than group X.
Then a new predictor which includes pairwise terms, trained on a corpora of group X and Y, will work out to be:
pred = a*written_evaluation + 0.33*written_evaluation*isY + other_factors
Interestingly, everyone's favorite bogeyman, namely redundant encoding ( http://deliprao.com/archives/129 ) will actually help fix this problem *even if you don't include the biasing factors in the model.
How do you find out that the written evaluation is biased "to the tune of 25% against group Y?"
THAT is the problem. It's not obvious to me how you would go about determining written evaluations are biased (and to what extent!) against group Y without somehow relying on subjective societal values that are different everywhere and continually evolving.
You build a sufficiently expressive statistical model and include the potentially biasing factors as features in the model. Then the model will correct the bias all by itself because correcting for bias maximizes accuracy.
In the example above, you find the bias by doing linear regression and including (written_evaluation x isY) as a term. Least squares will handle the rest. If you using something fancier than least squares (e.g. deep neural networks, SVMs with interesting kernels), you probably don't even need to explicitly include potentially debiasing terms - the model will do it for you.
I give toy examples (designed to illustrate the point and also be easy to understand) here: https://www.chrisstucchio.com/blog/2016/alien_intelligences_...
This paper does the same thing - it discovers that standard predictors of college performance (grades, GPA) are biased in favor of blacks and men, against Asians and women, and the model itself fixes these biases: http://ftp.iza.org/dp8733.pdf
Statistics turns fixing racism into a math problem.
If the topic were anything less emotionally charged, you wouldn't even think twice about it. If I suggested including `isMobile`, `isDesktop` and `isTablet` as features in an ad-targeting algorithm to deal with the fact that users on mobile and desktop browse differently, you'd yawn.
Who decides what the "potentially biasing factors" are? How is that decided without somehow relying on subjective societal values?
Factors that no one thought were biased in the past are considered biased today; factors that no one thinks are biased today may be considered biased in the future; and factors that you and I consider biased today may not be considered biased by people in other parts of the world. I don't know how one would go about finding those "potentially biasing factors" without relying on subjective societal values that are different everywhere and always evolving.
Go read the wikipedia article on the topic: https://en.wikipedia.org/wiki/Omitted-variable_bias
It's true that as we learn more things we discover new predictive factors. That doesn't make them subjective. A lung cancer model that excludes smoking is not subjective, it's just wrong. And the way to fix the model is to add smoking as a feature and re-run your regression.
Again, would you make the same argument you just made if I said I had an accurate ad-targeting model?
Many people today would object a priori to businesses using race as a factor to predict loan default risk, regardless of whether doing that makes the predictions more accurate or not. In many cases, using race as a factor WILL get you in trouble with the law (e.g., redlining is illegal in the US).
Please tell me, how would you predict what factors society will find objectionable in the future (like race today)?
I claimed a paperclip maximizer will maximize paperclips, I didn't claim a paperclip maximizer will actually determine that the descendants of it's creators really wanted it to really maximize sticky tape.
Now, if you want an algorithm not to use race as a factor, that's also a math problem. Just don't use race as an input and you've solved it. But if you refuse to use race and race is important, then you can't get an optimal outcome. The world simply won't allow you to have everything you want.
A fundamental flaw in modern left wing thought is that it rejects analytical philosophy. Analytical philosophy requires us to think about our tradeoffs carefully - e.g., how many unqualified employees is racial diversity worth? How many bad loans should we make in order to have racial equity?
These are uncomfortable questions - google how angry the phrase "lowering the bar" makes left wing types. If you have an answer to these questions you can simply encode it into the objective function of your ML system and get what you want.
Modern left wing thought refuses to answer these questions and simply takes a religious belief that multiple different objective functions are simultaneously maximizable. But then machine learning systems come along, maximize one objective, and the others aren't maximized. In much the same way, faith healing doesn't work.
The solution here is to actually answer the uncomfortable questions and come up with a coherent ideology, not to double down on faith and declare reality to be "biased".
Are biases distinct from "preferences" - humans view flowers as more pleasurable than insects - human language associates flowers with pleasurable terms, states and so-forth.
"Bias" is term associated with "irrational beliefs" whereas "preferences" more often imply "arbitrary preferences". Especially, biases are held to prevent rational deduction whereas preferences have no such stumbling block.
Now, one supposes that question would come down to whether a computer would "know it's a computer, not a person".
If the AI was asked "do you like cockroaches or daisies better", would it say "why daises are prettier and smell better" or would it say "most people like daisies but I'm a machine, can't smell or taste, and only care about the preferences entered into my control panel" (or something).
And you'd expect that a thing that merely "parroted" human speech without understanding would give the former answer.
Which is to say I don't think you are really fully grappling with word-association and word-logic coming together, ie, "meaning".
I thought the section on "Challenges" could have been stronger. You talk about the bias in "the basic representation of knowledge" used in these systems today -- but it's not like there isn't aren't other possible representations of knowledge. How much effort has gone into exploring knowledge representation (and approaches to derive semantics) that are designed to highlight and reduce biases look like?