Under this definition of 'bias', an unbiased model would, say, spit out equal associations between any occupation and any gender/sex/age/race/religion label.
We should probably ask ourselves whether that's a strictly desirable outcome, since by definition the 'biased' model has a higher predictive value. How much accuracy are we willing to sacrifice for the sake of erasing inconvenient facts about either our world, or our current models of the world?
We could all pretend that we all knew what "unbiased" was, as long as we lacked mechanisms for putting numbers on these things. We could meet our fuzzy conception of the biases with our fuzzy conceptions of what "unbiased" ought to be and we could all speak back and forth to each other praising the virtues of how unbiased our models of the universe are, and there was no way to dig in any farther to see if we were all actually saying the same thing.
Now we can put numbers on it. So, do it. What's the goal? Be clear. It's a program. Anything you can clearly specify can be done. It's not useful from an engineering perspective to define the goals entirely by examining each model in sequence and deciding on the spot that it's not "unbiased"; give us a definition in advance, so we can build a model to try to fit it.
Once the goal is stated clearly, the engineering work becomes much easier. We can even iterate on the goals. I'm not asking for perfection on that front on the first try any more than I ask for it anywhere else. But there's been enough of these articles that just point at a specific point in the space of models and call that point problematic. Move on the to next phase. What wouldn't be problematic? Exactly? At least take a stab at it instead of trying to make vague insinuations so we can iterate on the stab.
Nor am I trying to prejudge what those answers will be. It isn't necessarily the case that the only possible answer is to just slam in 50/50 numbers for the genders (or whatever other constant values you want for whatever other genders you want). Though if that is what you want, do it, and see what happens, and iterate from there. But what other things can be tried too?
The proper answer would be a series of clarifying questions or clearly specifying the qualifications of the answer. (E.g. geography, general medical practitioners)
Here's an example detached from any sort of politics. If you are training a language model, the maximum-likelihood model is one where the frequency of every word is the frequency that you observed that word, and the frequency of unseen words is 0.
On your training data, you'll measure the maximum-likelihood model as making more accurate predictions than any other model. But it's also useless, because when you use it on new data, it assigns an impossible probability, 0, to any sentence containing a word it's never seen.
This is a bias. You can't correct it by getting more data. You correct it by being an intelligent human who knows the maximum-likelihood model is wrong, and applying a correction on top of it (smoothing).
Now account for the bias (observed by Arvind Narayanan) that all your training data is in the past, and all the predictions you want to make are in the future, but circumstances change between the past and the future.
That's the entire point.
You can't teach a model not to reduce probabilities to zero just by showing it more data. There will always be more unseen data. And you, a human, know that there will always be more data. So you apply smoothing to the model so that your model isn't nonsense.
I'm giving an example of a bias that you can't solve with data, that you must solve with design.
Any word level model trained on any dataset will say it is zero. But that is incorrect; humans know that there is always a probability that some arbitrary sequence of characters will suddenly appear in text (think a GUID in technical documentation). To account for this, the model's author has to make sure to smooth the output so it doesn't produce these zero probabilities.
No amount of data can ever fix this problem (although more data can put better bounds on the smoothing factor).
>>> the maximum-likelihood model is one where the frequency of every word is the frequency that you observed that word, and the frequency of unseen words is 0
>> I think that's false. A model may never reduce any probability to zero. Novel data is a normal feature of modelling.
> But there are better ways to build a model.
Yes of course there are. That's exactly what the OP said ("You correct it by being an intelligent human who knows the maximum-likelihood model is wrong, and applying a correction on top of it") but you seemed to be arguing against it.
(And to be clear, there are also alternatives to the maximum likelihood model)
https://en.wikipedia.org/wiki/Training,_test,_and_validation...
Even with test/train splits you still need to smooth your model's predictions because it will always get unseen words. That's just the nature of word based lanugage models.
Because test data is unseen at training time, it cannot possibly affect the way your model is trained. The test data is not how you prevent overfitting, it's how you measure overfitting. To prevent overfitting, you need to design features such as smoothing into your model.
You seemed to be arguing that only way to know the model is wrong is to insert a human who just knows a priori that the model is wrong and what "correction" to apply. But I think this is a false analysis. You can measure a model's accuracy by seeing how well it predicts data outside the training set. If the model is highly predictive, you have a good model. It doesn't take a human's subjective analysis or principled intervention to determine whether the model is accurate with respect to the available data.
This article concerns cases where the model is admittedly accurate (with respect to the available data), but is subjectively considered objectionable. That is a totally different issue.
So models must not be "one size fits all", but should allow a measure of personalization, so that in a world where most people prefer dogs to cats, cat lovers can still get content they enjoy.
Sure, you can build a discriminating classifier or generative model that is the most accurate, correctly identifying / emulating the reality of our world down to 5 nines; and nobody says you shouldn't be able to identify or quantify all of these "inconvenient" associations. The trouble is always when you intend to make decisions based on your framework -- the decision to offer somebody a loan, the choice of language your chatbot uses, etc. -- and that is where fairness ought to be paramount.
And yes, if you manage to build a discriminating model that sneaks in protected classes through indirect causal effects with no attempt to suppress them, it would yield your insurance agency higher returns over time, and just because that's the way the world is, currently. But all this would achieve is perpetuating the current status quo, placing short-term gain ahead of long-term equality. You might be doing right by yourself, but that still makes you morally impugnable.
One shouldn't need to wait for an overreaching law prohibiting such indirection, to do the right thing.
How far do you expect systems to go to ignore real-world associations? What if it is coming down to personal safety? Would you object to a self-driving car that routes itself around more dangerous neighborhoods? What about a model that predicts that unknown men on the street at night are more dangerous than unknown women?
Imagine a simple system where people are only evaluated for credit cards and mortgages, and businesses can only offer one of these services. Some people are systematically denied credit cards. Now they have to have more cash for daily needs, which makes them objectively worse as home owners, so the mortgage loaners pick up on this fact and start systematically denying them mortgages.
There's no business opportunity here. For either lender, the prospects in this group are worse, only because the other lenders think so. Breaking the barricade is much harder than just starting a new lender or either type.
Three strength of the effect varies but is hard to measure. Not impossible though.
Afaik the black families who were segregated out of their neighborhoods back in the 1940s, 1950s and going into the 1960s were never provided with another comparable "business opportunity" that could have "righted" their situation, they had to settle with living in what turned out to become "ghettos" (for lack of a better word).
> Would you object to a self-driving car that routes itself around more dangerous neighborhoods? What about a model that predicts that unknown men on the street at night are more dangerous than unknown women?
Following the same line of thought, would you be ok with an AI system giving a person named Deion or Jayla or Latisha a higher interest rate on their mortgage (and so, potentially, driving them out of certain markets) compared to the interest rate offered to persons named Chad or Emma or Sophia?
If it's your money that you're choosing to lend (or not lend), would you ignore that information or take it into account when making a decision if you should offer that loan and how much the rate should be adjusted to cover the risk of default? What if it's not your money to do with it as you please, but money entrusted to you to invest for maximum results?
Sure, if you knew what this particular person is actually like, then you'd base the decision on better data than just their name, and it could well be that this Jayla would get a much better deal than this Sophia; but even if you had a perfectly fair system that correctly estimates the individual risk of default without prejudice causing worthy individuals being dragged down by association with some group, then there would still be a correlation with the name simply because in the current reality the proportion of "risky Jaylas" is larger than the proportion of "risky Sophias".
> Following the same line of thought, would you be ok with an AI system...
You didn't answer my question. I asked it because I want to know if the people most vocally arguing that de-biasing is a moral imperative will admit even one case where there might be a compelling reason to see the world as it is, even if that association is considered "problematic".
If people will admit this, then we can argue over where the line should be. But many don't appear to admit that a line even exists.
I will admit that the line exists, and that an incident like this is a clear example where removing hurtful associations is proper. This was a case where a ML model reinforced a loaded and racist stereotype, and the harm of removing that from the model is almost zero: https://www.theverge.com/2015/7/1/8880363/google-apologizes-...
I didn't find those questions to be that smart, to be honest, they're more on the scare-mongering side, and I find that a smart question should have half of the answer included in it and a scare-mongering question doesn't look like it contains anything smart (at least to me). But if you really want to know my answer is "yes" to all of your questions. To get into more details: I grew up as a kid into a middle-class-ish family and back then I had no issues going to the "ghetto"/dangerous area of the town I grew up in (I grew up in Eastern Europe, so that the "dangerous" area was populated by the local gipsy community instead of the AfroAmerican/Latino communities now associated with the "dangerous" areas of US cities). I turned up fine.
> If people will admit this, then we can argue over where the line should be. But many don't appear to admit that a line even exists.
I know of that line, I was just trying to say that further reinforcing it using ML techniques will only aggravate things at a societal level (so that "dangerous" areas will become even more "dangerous"). At least when the discrimination happens out of our (us, humans) own volition there are ways to fix it, but once we "outsource" our racist-tendencies to ML-like tools the voice inside of us telling us that this is all wrong will become even weaker. After all, the algorithms/machines are more "right" then us, humans, that's what we like to think.
Three operational principle of the capitalist business is extracting value from a customer or an externality. That is from the outset unfair. It is just, as long as businesses act independently.
Your statement implies that you would support this.
These businesses were using mental models that had a high predictive value. Do you feel that it was worth it to "sacrifice this accuracy" with civil rights laws?
By that logic, there'd be no business catering to poor people at all.
The businesses that do cater to the poor classes exploit them. See fast food restaurants, payday loan lenders, casinos, etc. None of these businesses exist to improve people's lives, but exploit a weakness at a particular moment.
Let's try again: If there is a business that is at capacity, is it appropriate for it to use racial criteria that may have some ability to predict how much a person will spend, if this increases profitability? Is not doing this "erasing inconvenient facts"?
Some intro reading for the tip of the iceberg:
https://en.wikipedia.org/wiki/Redlining
https://en.wikipedia.org/wiki/Blockbusting
https://en.wikipedia.org/wiki/Montgomery_bus_boycott
https://en.wikipedia.org/wiki/Sit-in
Also, you seem to be implying capitalists aren't authoritarians. But that certainly isn't self-evident in the context of what I replied to.
Context isn't something you impose in the middle of a conversation; it's something you should evaluate before jumping in.
If (unsophisticated) AIs were in charge, the civil rights movement might never have happened. Black Americans were the victims of institutionalized racism. That racism was self reinforcing: Society did not support the education of blacks, which caused them to be uneducated, which reinforced the stereotype that they were stupid, therefore making it obvious that educating them is pointless, which caused society to not support the education of blacks.
How would an AI controlled system reason to understand that the rules of society itself are creating the conditions of inequality? Will it merely strengthen positive feedback loops, reinforcing stereotypes and preventing reform?