Google Translate is inherently biased – An example of an Hungarian translation
reddit.com
reddit.com
The proffered solution is to use singular "they" of course. Trouble is, people don't talk like that and so from a language-is-as-it-used perspective (descriptivist), always using singular "they" makes your translation worse.
I would offer a solution like this: allow the person using the translation program to specify pronoun rules.
There is a school of thought that says we should change how we use the English language. This school thinks that the proper way to speak is to assume non-binary until informed otherwise and that appealing to "what sounds right" or "the rules of grammar" is unacceptable linguistic conservativism.
I am not making that argument. I'm just raising it because it seems relevant here. I would point out that, even if you disagree with this argument, it seems reasonable to think that people should be able to choose how they use pronouns, whether their usage flies in the face of linguistic convention/rules of grammar or goes along with them.
That said, precisely because communication is a mutual endeavour, and language is a coöperative framework, there is no guarantee that -- just because you choose to use "they" in this way -- others will follow. I wouldn't be terribly surprised if, within a generation, "they" became an unmarked definite gender-neutral pronoun in English (today it is arguably unmarked in the case of an indefinite antecedent, but extremely marked otherwise). But I wouldn't be terribly surprised if it didn't, either. Language change is an emergent process and not something that can easily be forced in either conservative or radical directions.
That's subjective.
> allow the person using the translation program to specify pronoun rules
"Translation is the communication of the meaning of a source–language text by means of an equivalent target-language text" (https://en.wikipedia.org/wiki/Translation)
Allowing the end-user to customize the translation according to their own personal biases introduces an entirely different set of issues. We should no longer call it a "translation" at that point.
I don't follow. There is an argument out there that we should stop using he/she and default to they (unless informed otherwise). Am I wrong about that? I didn't say I agree with it, just that it is a proposed solution.
> Allowing the end-user to customize the translation according to their own personal biases introduces an entirely different set of issues...
Yes, and allowing people to speak introduces this exact same issue...
It's not the purpose of a translation service - to let people customize the translation such that it fits their own personal biases and sits more comfortably with their world views.
A translation service requires user input, spoken or written, which is necessarily biased based on how the user speaks/writes/thinks. The service is also biased based on the data and methods that are used to engineer it.
In my opinion, it is perfectly fine for people to be biased. Whenever I choose to write, I am expressing my personal (biased) view. But when it comes to machine translation, I think it's reasonable to object to bias and so, for contentious issues like personal pronouns, I think it makes sense to allow people to choose.
In other words: there is simply no right way to decide between defaulting to "he", "she", or "they" for third person gender neutral. I use "he" (it's how I was taught and it sounds better to me). Some people would accuse me of phallocentrism or linguistic conservatism, but fine. In modern philosophy, "she" is used. In some circles, "they" is used. I see no issue allowing the user to specify their preferred rule, though it might be technically challenging (I have no idea).
And I can guarantee you've never used this line of reasoning against translation services performing spelling or grammar checking. Yet those are also "not the purpose of a translation service".
It's always interesting to see the twists people will go to try to use post-hoc justify an opinion.
"Translations are gender-specific. LEARN MORE"
and both versions
"he is beautiful"
"she is beautiful"
In the past this wasn't the case, AFAIR.
I guess they couldn't just patch a workaround for everything.
I'm also unsure if they should: let's say you want to translate a bit from the gospel "He broke the bread... gave it to His disciples... and He said..".
If you're translating from hungarian, would you have this translated as "he/she" ? It seems more correct to use the original "He" since you can infer from context what it is about.
At the same time, it's clear there is always bias in the training material, and probably "recover original text" is useless.
The HN title "Google translate is biased" is misleading, but it's a fascinating example of how AI can encode bias.
It probably doesn't matter if I'm using Google Translate to make hotel reservations when traveling. It matters a lot if you're going to use AI to scan resumes.
An engineer builds a chat protocol and app on top of UDP. Chat messages between users frequently never arrive and are lost.
When questioned about it, the engineer says "it's not my app that causing the message loss it's UDP that's causing it."
The engineer selected the communication protocol.
To avoid packet loss, I build retry logic into my app, or build on top of TCP.
What's the "I used TCP instead of UDP" equivalent of a dataset that can be used to build a perfectly non-biased translation service?
Your comment seems to suggest that everybody seems to deliberately choose the wrong datasets which result in racist and sexist products. What should we use instead then?
If the underlying dataset has issues, build a layer on top to address those issues.
"No", to both questions.
These systems are a work-in-progress, constantly improving. They have value, even though they are far from perfect.
Those criteria also align roughly with the acceptability of singular "they" in modern British English: a sentence like "everyone should mind their own business" is totally normal and standard and always has been; a sentence like "the unidentified miscreant left their cup on the table" will seem a bit strange to some people; a sentence like "Chris ate their lunch" will seem strange to a lot of people.
I wouldn't expect Google Translate to be capable of such cleverness so a simpler solution would be needed.
“correct translation” is a matter of taste. You’re creating a new work, after all, so there’s some authorial contribution.
So, bias no matter what then?
I wonder what's the point behind this entire Reddit thread then. More fuel for the fire?
I'm reminded of one of the early "oh, wow" moments with word2vec (a word embedding scheme), where the statement "King - Man + Woman" evaluates to "Queen". This is not a measure that the system understands anything, just that there are really strong statistical correlations that are preserved in the embedding that let you see how close vectors are to one another.
Necessarily, this sort of vector similarity or word choice probability is a function of (at least) your training data. If you want to stop over-weighting pronoun choice for adjectives like "pretty" and "weak" or "emotional" to be feminine, be more choosy about your training set and don't just snarf up reddit willy-nilly and go brrrrr.
That's a much more interesting (and complex) conversation.
Siphoning raw Reddit/Wikipedia is the closest you'll get to an honest snapshot of our internet-based society (for all of its tremendous flaws).
One could make the case that we don't need an "honest snapshot of our internet-based society" to build a Translation app. But that's debatable...
Being "more choosy" will inherently introduce a different set of biases. Perhaps those biases align more closely with our core beliefs and values - but they serve as bias nontheless.
One thing that will absolutely raise some people's hackles is pointing out that "intentionally modulating biases in translation techniques" already has a name, and it is "being prescriptive about language usage". I've found that generally, tech/progressive/machine learning ethics folks tend to believe that prescriptivism is the domain of, for example, stuffy old people like Strunk and White, how dare they tell me what is "proper" -- and then turn right around and do just the same with a straight face but don't seem to really realize that that's what they're doing.
At any rate, designers of such systems do need to face head on the fact that machine translation systems at scale both 1. need to reflect usage in the world in a descriptive enough way as to be useful and honest and provide coherent approximations of translation, even sometimes if it's reflecting perhaps unpleasant truths about ourselves we might not like to think about and 2. need to realize that their participation and choices of embeddings, etc., contribute to the corpus of texts in the world as well as reflect them, so their prescriptive choices are also real and will have feedback effects.
(2020) "A Scalable Approach to Reducing Gender Bias in Google Translate"
https://ai.googleblog.com/2020/04/a-scalable-approach-to-red...
"We also devised a new method of evaluation, named bias reduction, which measures the relative reduction of bias between the new translation system and the existing system. Here “bias” is defined as making a gender choice in the translation that is unspecified in the source. For example, if the current system is biased 90% of the time and the new system is biased 45% of the time, this results in a 50% relative bias reduction. Using this metric, the new approach results in a bias reduction of ≥90% for translations from Hungarian, Finnish and Persian-to-English. The bias reduction of the existing Turkish-to-English system improved from 60% to 95% with the new approach. Our system triggers gender-specific translations with an average precision of 97% (i.e., when we decide to show gender-specific translations we’re right 97% of the time)."
+
(2018) "Providing Gender-Specific Translations in Google Translate"
https://ai.googleblog.com/2018/12/providing-gender-specific-...
"Detecting Gender-Neutral Queries Many Turkish sentences that refer to people are gender-neutral, but not all are. Detecting which queries are eligible for gender-specific translations is a hard problem because Turkish is morphologically complex, meaning that reference to a person can either be explicit with a gender-neutral pronoun (e.g. O, Ona) or implicitly encoded. For example, the sentence “Biliyor mu?” has no explicit gender-neutral pronoun but can be translated as either “Does she know?” or “Does he know?”. This complexity means that we cannot use a simple list of gender-neutral pronouns to detect gender-neutral Turkish queries and need a machine-learned system. We estimate that approximately 10% of Turkish Translate queries are ambiguous, and eligible for both feminine and masculine translations."
The gender of the subject is derived from context.
The idea that men and women are the same, is one not borne out by facts and evidence, yet it’s still being pushed by extremists who ignore the science.
The suggestion that differences in men and women are a result of sexism is not borne out by facts and evidence either.
Rather than trying to force google translate to lie (Suggest that men and women are the same), perhaps we should just accept the truth - eg that most Plumbers are men.
Why do you think women don’t want to be plumbers? Do you think it’s because society decided plumbers are men, lack of female plumbing role models, because plumbing customers are all sexist and only employ men, OR because women aren’t generally that interested in plumbing...
Why do you think in countries where they have tried the most to get them to be plumbers, women have rejected it even more?
Why do you think women should want to be plumbers?
Men and women are extremely different in so many different ways, which should be celebrated, not denied.
Obviously anyone male or female who wants to be a plumber should be encouraged. But we should also freely acknowledge that 95%+ of plumbers are men, and that isn’t a ‘problem’ to be fixed.
Please demonstrate it then. Which research? Nothing you said demonstrates what you suggest.
"Results provide strong support for hormonal influences on interest in occupations characterized by working with Things versus People." - Gendered Occupational Interests: Prenatal Androgen Effects on Psychological Orientation to Things Versus People (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3166361/)
"Gender differences in personality tend to be larger in gender‐egalitarian societies than in gender‐inegalitarian societies, a finding that contradicts social role theory but is consistent with evolutionary, attributional, and social comparison theories. In contrast, gender differences in interests appear to be consistent across cultures and over time, a finding that suggests possible biologic influences." - Gender Differences in Personality and Interests: When, Where, and Why?" (https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1751-9004....)
It would be ignorant to assume that biology has zero impact on occupational interest.
The sexes evolved to share the workload, and specialise in different tasks. Men have greater spacial awareness so they can hunt better. Women have greater empathy so they can care for babies emotional wellbeing.
The “men and women are the same” lie is a very very recent ideological movement, and incredibly damaging.
Are you seriously suggesting this is down to ‘society’ telling boys they’re allowed to be plumbers and girls that they’re not allowed to be?
As I said, in countries where they have gone the furthest to try to eradicate ‘gender roles etc’, they saw the % of women plumbers GO DOWN. not up. Which would strongly suggest that it’s got nothing to do with ‘gender roles’
If you’re interested in looking at the research, go for it. There’s absolutely tons of extremely established studies into the differences between men and women, their different strengths and weaknesses, and why they decide to do different things in life, go into different careers etc (Unless it’s been cancelled or burnt, which is entirely possible)
[1] Why are There so Few Female Computer Scientists (Ellen Spertus, 1991)
https://dspace.mit.edu/handle/1721.1/7040
[2] How to Encourage Women in Linux (Val Henson, 2002)
https://tldp.org/HOWTO/Encourage-Women-Linux-HOWTO/
[3] What Happens to Us Does Not Happen to Most of You (Kathryn S. McKinley, 2018)
https://www.sigarch.org/what-happens-to-us-does-not-happen-t...
[4] Unlocking the Clubhouse: Women in Computing (Jane Margolis and Allan Fisher, 2001)
https://www.amazon.com/Unlocking-Clubhouse-Women-Computing-P...
Talking about gender roles on HN is odd because programmers tend to think so binary. Gender roles in modern society are informed by biology inasmuch as La Croix's flavor is informed by fruit but all you get in the discussion is "see gender roles aren't entirely arbitrary so everything about the status quo must be justified."
A non men/women example is neighborhood crime rates. A totally "unbiased" record of crime rates will show that black neighborhoods have on-average crime rates. But that ignores the fact that less policing is done in areas with lower crime rates and if you don't look you won't find it.
Theres nothing special here. Its human pattern recognition over fitting fishing for attention and a stamp on their simp card using political hot topics.
Ő takarít is translated as he cleans.
Ő takarító is translated as she is cleaning (the correct translation here would be she's a cleaner, also ő egy takarító would be the most used way...the original sentence was clearly modified to invoke controversy).
Guess which one the Reddit example contains from the two..also it was modified on purpuse, you can see from the sentences.
Is your point that there be disagreement on ML policy? I don't think that makes any difference to the fundamental research problem.
That's why people who experiment with it themselves don't see as strong bias.