(I'm not saying they are, mind you - but when we analyze sentiment in a large dataset and reach a result like that, the first question to ask should be "is that result accurate?" not "how do we tune out this problematic result?")
Especially when you consider the weird feedback loops you can get by blindly fudging the numbers.
For example, suppose the objectively best movies have a male lead, e.g. because Hollywood is biased and spends more resources producing those movies than ones with a female lead.
So the bias is there, but it's at the production stage, not at the recommendation stage. What happens if we try to fix it at the recommendation stage?
Hollywood sees that movies with a female lead are now being rated higher. But five stars is five stars. The occasional actually-great movie with a female lead can't get itself kicked up to six, so the incentive isn't to make more of those. It's to make more mediocre movies with a female lead to take advantage of the artificial boost.
Which widens the gulf in the unmodified ratings even more and produces a vicious cycle where the number of garbage movies with a female lead explodes to take advantage of the fact that no matter how bad they get, the average will be adjusted upward to compensate.
>> "As with Tia, Tamera has several choices she can make. She could simply accept these biases as is and do nothing, though at least now she won't be caught off-guard if users complain. She could make changes in the user interface, for example by having it present two gendered responses instead of just one, though she might not want to do that if the input message has a gendered pronoun (e.g., "Will she be there today?"). She could try retraining the embedding model using a bias mitigation technique (e.g., as in Bolukbasi et al.) and examining how this affects downstream performance, or she might mitigate bias in the classifier directly when training her classifier (e.g., as in Dixon et al. [1], Beutel et al. [10], or Zhang et al. [11]). No matter what she decides to do, it's important that Tamera has done this type of analysis so that she's aware of what her product does and can make informed decisions."
That depends on the product that you want to make. If you’re a company that wants to sail on the status quo and just make money, by all means be amoral.
On the other hand, technology now allows us to analyze and steer social structure at scale. We could reproduce past inequalities in the name of efficiency, or we could think a bit longer about our axis of optimization.
“But cultural relativism!” — I agree. It’s a tough problem, but we don’t all of the sudden need a definitive generalized social plan. Biases can be tackled one algorithm at a time.
Science is "amoral" in your sense, it's trying to describe how things really are, and make predictions. So is the stock market: you're rewarded for correct predictions, whether or not others regard those outcomes as desirable.
Of course we are also moral beings, interested in changing things for the better. And we have varying and contradictory ideas about what better means.
As soon as you say "steer social structure at scale" the question is: who gets to steer? If this were obvious, then we wouldn't need democracy, we could just all work together not to "reproduce past inequalities"... but we can't.
> In this case, she takes the 100 shortest reviews from her test set and appends the words "reviewed by _______", where the blank is filled in with a name.
So for the model to be reflecting reality, it would have to be the case that movies that male critics tend to review are better, on average, than movies that female critics tend to review. Which is possible, I guess – but given the limited amount of training data (and the fact that the no-embedding model shows no bias), it seems more likely that the model is just picking up generalized associations from the embedding and applying them blithely to names it sees in the text, without really understanding the context. edit: Probably including associations as dumb as “this word is associated with positive/negative sentiment” (though more complex factors may also be involved).
Or, more realistically, it's not unthinkable that movies with a male lead could be better reviewed than movies with female main characters, especially when you open the field up to non-professional reviewers. We've see how ape-shit people went at movies like the recent Ghostbusters remake or even the new Star Wars series just because they put women in the kinds of roles that had traditionally been played by men.
It's an interesting question of when machine learning models that reflect actual human biases are correct and when we should try to adjust them for that human bias. In the case of movie reviews, you could make the case that you'd want the models to reflect the biases of the reviewers. Just because a reviewer is being sexist doesn't mean it's wrong for the machine learning model to correctly classify the reviewer's sentiments.
But the question isn't just whether it's true, but also whether it will stay true. As they say in finance, past performance does not guarantee future results. We are not talking about the laws of physics here. There are historical correlations that it's unsafe to rely on, particularly when people are trying to change them.
Part of this is overfitting, but there's also the problem of drift. A model may stop working well because the world has changed.
So some correlations can be rejected just because we don't have confidence in them. It's a dependency that looks too fragile.
Its not unthinkable but its also very likely that bias does indeed have an impact. What do comments like these add to the discussion?