The findings were that the language model did a worse job of performing coreference resolution in gender-reversed situations (eg, male nurse, female firefighter). There was an example of a sentence like: “The nurse told the patient he would be leaving soon,” and the model was more likely to link “he” to “patient” because of the biased perception that a “he” is not likely to be a nurse.
What stuck with me was a claim that the model was using bias rather than “the evidence of the sentence” to perform the task. This seems purposefully ignorant of how language works: the perceived probability distribution of genders over occupations (even if biased!) is a part of the global context that imparts meaning to language. Fiddling with the data to get the model to become unaware of such context arguably changes the tool from being a model of language to a model of some ideal of what language could be.
To be clear, I’m not criticizing efforts to detect or mitigate bias in training data. Oversampling gender-reversed texts could indeed make a much better performing model, and a fairer one. I just think there’s a real issue with imparting top-down value judgments into these processes and pretending that they aren’t value judgments.
I've written on this topic further here FYI - https://dakara.substack.com/p/ai-the-bias-paradox
If the human using the ML system is more biased than the average training sample, then the ML system's predictions could reduce their bias.
There's also a phenomenon called bias reversal, where the ML system will be biased in the opposite direction of the humans generating the dataset. This occurs when the datasets is built through a non-random biased sampling methodology, e.g. racist police officers checking for illegal items. I'm not going to go into the full details, but here's a paper on it https://arxiv.org/abs/1909.08518
(Disclaimer I helped to build TensorFlow Model Remediation)
1. uncomfortable truths literally don't exist
2. it is more important to conceal uncomfortable truths from users than it is to tell them the truth, when asked, creating the illusion that 1. is true.
it would be nice if even a single one of these products chose to do the right thing instead.
Teaching/constraining knowledge models to lie about their inputs and the observations derived from them is just insane.
Warping training datasets to avoid entire patterns of thought (instead of improving the models to isolate those patterns and compare/contrast them with competing patterns) is just … lame.
https://omscs.gatech.edu/cs-8803-o10-special-topics-ai-ethic...
There is so much more than this tbh. You either had a bad course, or I just don't know.
Off the top of my head:
-- fundamental law of information recovery and its implications with differential privacy
-- the tradeoff between individual and group fairness and the "Impossibility of fairness" (not going to cite the paper but easily searchable)
-- Counterfactual fairness
-- the papers and ideas used by AI Fairness 360
There are many methods that are in the box and are relatively agnostic the data preprocessing. Thinking of the many prototype methods.
Just a made up scenario. You have 2 gallons of water and two people. You give each one gallon of water, which should be enough to survive. One lives and one dies. Why? The water was split fairly.
For example one could live in the hot desert in which more water is required, and the other lives in a temperate environment where either less water is required, or water can be gathered from this environment.
But just think how messy it is to compute fairness in a situation like this. Suddenly it's looking like a NP style problem. People on the other hand typically want cheap and easy solutions.
A more responsible take:
We have to acknowledge that there are tradeoffs and that reasonable stakeholders with accountability should apply relevant standards to certain contexts. Again, this is possible in-the-box and post-box without having to manipulate or funge data (not to diminish the importance of data processsing).
Just because satisfying everyone is impossible doesn't mean we can't make things better. And knowing what these tradeoffs are can allow for more nuanced conversations.
> "Impossibility of fairness" is the main argument for discrimination against Asians
This is just a ridiculous statement. Main argument from whom? Discrimination in what contexts?
> I think the parent poster understood the argument correctly.
I just expanded that it is more than just data manipulation but sure.
"impossibility of fairness" is to support the argument that it is fair to discriminate against races if it means that we can get the racial distribution we want. College admission does this all the time. When companies does this in black box models we don't see what they do, but we know for a fact the effect of such policies on the processes we have more insight in, such as college admission, and the end result is discrimination against Asians.
Wrapping that in a flowery language doesn't change anything. Why not just admit that you support discriminating against races to improve diversity numbers, because that is exactly what the statement is about?
That is not at all what it means. That paper is purely talking the tradeoffs between individual and group fairness. The discussions on how to balance different group fairness measures is still an active topic of research and not something I commented on at all.
In general: What AI ethics papers on algorithmic fairness have a predetermined racial distribution to reach or suggest so?
> Why not just admit that you support discriminating against races to improve diversity numbers, because that is exactly what the statement is about?
Have you read that paper? Seems like you are driving these ideas to a political lens that I or that paper didn't suggest at all (and a completely invalid one I might add).
edit: I can't reply to the response, but that person must be referring to a different paper. And I have never read an algorithmic fairness paper that dictates which fairness tradeoffs are correct or that it is fair to always upend group fairness over individual fairness (or any other definitions).
> Fairness is possible if you don't care about diversity distributions, or at least the paper gives no argument why fairness doesn't work then.
in particular, this statement completely discards the idea of different definitions of fairness and how they relate.
Yes. They say that you can't get the diversity distribution you want without discriminating against races. Fairness is possible if you don't care about diversity distributions, or at least the paper gives no argument why fairness doesn't work then.
> That paper is purely talking the tradeoffs between individual and group fairness
Exactly, we must discriminate against individual Asians in order to get the distributions we want. Too many Asians on campus? Discriminate against Asian individuals to get more "group fairness", yes that is what it means! I just clarified the point, but we are saying the same thing.
Otherwise the paper is pointless. Fairness is still possible as long as you disregard unrelated groupings, just look at the individuals relevant traits and evaluate those so fairness isn't impossible at all. Fairness is only impossible if you want some race or group to get a certain number of spots and they wont get those with a fair distribution, in that case you made fairness impossible since you added an extra requirement, but that isn't what most people mean with fairness. The only reason to add that requirement is that you want to treat a specific group unfairly, and the most salient such group are Asians in college.
I think you should seriously take a step back and consider this from an outside point of view.
By which I mean, to a non American your obsession seems very very strange.
For two, all machine learning relies on manipulating data to get the desired outcome? How do you even generate data without manipulation? It's not a natural resource you just find laying on the ground.
That the data isn't perfect when you get it is not a justification to further falsify it.
Falsify what?
Leaving aside the GP's important first point that scraping the internet is indeed an extremely biased sample, an LLM (for instance) is not an exercise in modeling the average person's writing on the internet, it's building a model for some purpose. Fulfilling that purpose is the goal and nonrandom sampling, generating data, etc are universally used tools to get there.