No Language Left Behind
ai.facebook.com
ai.facebook.com
That’s exactly right. There’s too much bias in society that if something isn’t perfect, then why bother? Nothing is perfect, so with that attitude there can be no progress. Thank you for doing important work!
You can check the list at https://en.wikipedia.org/wiki/List_of_languages_by_time_of_e...
To make an example and be blunt: I do not expect any European country official language to get extinct anytime during our lifespan unless that country gets destroyed, which obviously won't be a good thing.
As for brain implants, I won't hold my breath.
As for a European country official language (strange line to draw), Icelandic is already in a state where younger Icelandic natives speak to each other in English because smartphones do not support their native language. It is entirely possible that Icelandic will be in danger of extinction this century[2].
[1]: https://www.youtube.com/watch?v=t3qbYFvOHwk [2]: https://www.nytimes.com/2017/04/22/world/europe/iceland-icel...
It might take some work (and more media being produced in it, surely) but I don't think it's at risk
> because smartphones do not support their native language
Nowhere in the article says that. And most smartphones do even have support (including the Icelandic keyboard)
what is not supported is voice activated devices:
> "“Not being able to speak Icelandic to voice-activated fridges, interactive robots and similar devices would be yet another lost field,"
Regarding Icelandic, I know little about it except that it has a lot of non-ANSI if written properly. I already know many latin language dialect speakers that developed their behavior when the limits of SMS unicode were significantly smaller than ANSI, with long term repercussions. I'm not sure why facts like that would need to be in the article to discuss here?
> it's a bit of a mistake to say latin disappeared but I digress
Latin is a dead language. It's not extinct (because it's used for ceremonial reasons by the Catholic Church) but it has no native speakers. This is not a remotely controversial statement. Old Church Slavonic has a similar role in Slavic Orthodox Churches and is also a dead language for the same reason.
> Nowhere in the article says that. And most smartphones do even have support (including the Icelandic keyboard).
In my defense, I can't read the article -- I linked it because this Tom Scott video[1] uses that article as a reference for the Icelandic example he gave where he explicitly says that Icelandic is not supported by modern smartphones. My phone does have Icelandic support now so I guess the statement was only true at the time he said it?
You're of course right about "Latin is dead" (for practical purposes). But there's lots of nuances that get lost in this statement
Latin is dead in the same way as Middle English is dead. The Latin that died is the one around the late Roman Empire time that was "photographed" and frozen in time as Ecclesiastical Latin (oversimplification, of course)
And of course the modern romance languages derived from it, but there's no exact time where those "flipped the switch" and became modern French, Spanish, Catala, Portuguese, etc. So in a way it could be argued that they're Latin 3.0 w/ DLCs (which I'll totally give that it's a big stretch)
Would you feel the same way if that includes all languages you know, including all versions of English?
It's not just about becoming bilingual, a population becoming bilingual in a "prestige language" is the first stage of language death (though of course it doesn't /always/ lead to language death).
How would you feel if your kids only learned Chinese[1], and not a word of English?
[1] I’m assuming you’re not Chinese
It is true that social pressures kill off local languages, but it's usually not because parents don't want to teach their children their mother tongue, it's that people stop using the language to communicate because of the influence of the "prestige language". My parents (and all of the parents in the immigrant community I live in) went through great pains to teach their children their native language.
The burden is not in knowing another language, the burden is in making the "less useful" language your main one.
Your parents and all of the parents in the immigrant community you live in are very happy that English is now your main language which you acquired through school, friends, tv etc. Or maybe they put you in your-language-only schools, made sure you socialized with your-language-only friends and watched your-language-only media?
I speak a language where I've never seen any translation for it... and when translated manually, my mum totally butchers the meaning lol.
Either way, any work in this area is more than welcome, but damn it's a hard problem.
As well as in the research paper: https://research.facebook.com/publications/no-language-left-...
We're at a point where it's now possible to determine the shape of every language, provided there are enough speakers of the language left who are both able and willing to help.
<Snark> Once done, Facebook can then commodify their dissent, and sell it back to them in their native language. </Snark>
It seems to me that the happy accident of doing this research at the start of getting all human knowledge digitized is part of the unreasonable effectiveness of this overall technique.
Had it happened in 200 years, it might not have worked, right?
>Translating Wikipedia for everyone
Hmmm.
While there is very definitely utility in doing things like this, I do kinda fear "poisoning the well"-like effects of feeding (even partially-) AI-generated-data into extremely common AI-data-sources.
There's some info on it in a blog post[1] and the MediaWiki "Content translation" page[2], but does anyone know of any studies on the quality of the translations produced? I can absolutely see it being a huge time-saver for people who are essentially fluent in both (there's a lot of semi-mechanical drudgery in translating stuff like this that could be mostly eliminated)... but people are pretty darn good at choosing the easy option of trusting whatever they're given rather than being as careful as they should be. It kinda feels like it runs the risk of passively encouraging people to trust the machine's choice over their own, as long as it isn't obviously nonsense, and the cumulative effect could be rather large after a while.
[1]: https://diff.wikimedia.org/2021/11/16/content-translation-to...
A machine-translated Wikipedia would not be a trustworthy source of information at all, yet would look like one. I think that does significantly more harm than good.
[0] Suggestions for better alternatives welcomed.
(As an example, it would be absurd for lithuanian wikipedia to include sources in japanese - that would be not usable AND not usefull for the wikipedia readers, editors...)
TBH it just sounds like you've redefined the word "unique".
Full list of currently supported languages can be found here: https://github.com/facebookresearch/flores/tree/main/flores2...
I currently host the largest collection of bilingual Manx[0] <-> English texts (~1MM words). How would I formally get in contact to chat about the steps to make machine translation available (and would there be grant opportunities available for further production of machine-readable data?)
Regarding grants: we have offered compute grants previously with the Workshop on Machine Translation (last year: https://www.statmt.org/wmt21/flores-compute-grants.html, this year: https://statmt.org/wmt22/large-scale-multilingual-translatio...) and we have an RFP, but it's currently focused on African languages: https://ai.facebook.com/research/request-for-proposals/trans...
I am a former professional translator (Japanese to English) and am now supervising research at the University of Tokyo on the use of machine translation in second-language education. As I have written in a few papers and essays [1], advances in MT have raised serious questions for language teachers. The ready availability of MT today, including on Facebook and Instagram, means that language students use it a lot while studying. We don’t know yet, though, how that use of MT might affect our students’ acquisition of other languages or their motivation to keep studying those languages.
One of the hurdles educators and researchers face is finding out how MT is being used in the real world. Most education in modern languages is focused on giving students language skills that they will be able to use later in work, education, and daily life, and textbooks and other learning materials are typically shaped around real-world situations. We are now struggling to adapt those materials for the age of MT, because data on the actual use of MT is very hard to get.
Like Google, Microsoft, Baidu, DeepL, and others, Meta must have huge amounts of data on how your users are using MT to communicate. Any information and insights about that MT usage that you can share with world—just as you have generously shared your NLLB models—would be most welcome.
When I learned Spanish, I spent a lot of time chatting on Facebook with native speakers, and using Google Translate as "training wheels" to help me formulate sentences, and understand words and phrases I hadn't learned yet. It worked pretty well at the time (2012) except in cases of slang and typos that Google couldn't handle. I also used it a lot to help me translate blog posts from English to Spanish. Eventually, I graduated from the training wheels and was able to use Spanish fluently without the help of MT. More than once, while not using MT, I was told that I spoke Spanish with a "Google Translate accent", which I'm sure was more of a reference to my grammar than my accent, since my spoken practice was 100% with native speakers.
When I learned Hungarian (2019-now), at the beginning, Google Translate wasn't good enough to use for much more than getting a rough understanding of formal text, so I learned in a more traditional way at a school and with native speakers. Then the pandemic prevented me from doing both of those. I started chatting with native speakers on Facebook, but it was very difficult without MT and involved a lot of asking my conversation partners for translations and explanations. Progress was frustratingly slow. Then I discovered DeepL's MT, which was extremely good with Hungarian. I started using for chat conversations and emails, and people were shocked that I was managing to communicate with them so fluently. My progress of actually learning the language for myself accelerated dramatically. I've become conversational (B2/C1) in Hungarian in 2.5 years with very little in-person practice. Often, it takes native English speakers 5 years of in-person practice to reach that level. I'm convinced that MT played a key role in my ability to learn quickly.
When I use MT, I have a simple rule, that I have to understand each word of a translation before I send it. So I carefully read the translation, making sure that I understand each word. Sometimes that means I have to look up individual words/grammar before sending a message (I often use wiktionary for that, because it shows etymology), and other times, it means that I'll replace unfamiliar words or phrases in a translation with words and phrases from my own vocabulary. Over time I rely on MT less and less because my own vocabulary becomes stronger. I really believe that they key to learning a language quickly is to start USING the language as quickly as possible. Once you're using a language, your brain automatically starts picking up the skill. With traditional language learning, using a language can be very difficult in the beginning until you've reached a conversational level, but with MT, you can start using a language before you know everything.
For Spanish, I almost never use MT anymore. Sometimes I use it as a quick dictionary for an unfamiliar word, but my Spanish level is C2 and I use Spanish every day so it feels natural. I'm not ever translating in my head anymore.
For Hungarian, I'm still using MT often, but I don't need it during conversation (either written or spoken). Besides using it to translate things I don't know, I also find it useful for inputting Hungarian characters that are a pain to type with my US keyboard, and for conjugating words correctly when I know the root but am struggling for the correct ending. Often I'll know what I want to say in Hungarian, but I'll open DeepL and type in English, then adjust the translation to use the words I want before I copy and paste the Hungarian. I'm essentially using MT as a guide to help me craft my sentences even when I know what I want to say.
In summary, MT is awesome for language learning and for assisting language skills in development.
Most of the discussions I’ve had about MT have focused on language learning in school contexts. In Japan and most other countries (though often not in English-speaking countries), all children have to study at least one foreign language in school. As with all compulsory education, low motivation and poor study skills are a constant problem. In such contexts, MT seems to many teachers and students just to be a way to cheat on classwork. And since very few of today’s veteran teachers were able to use MT when we were young and studying languages, we don’t understand how we can guide even our motivated students on using it productively. I will be sure to share your insights with my colleagues.
A couple of comments:
> More than once, while not using MT, I was told that I spoke Spanish with a "Google Translate accent", which I'm sure was more of a reference to my grammar than my accent....
This can happen with traditional methods of language learning, too. The language of textbooks, like the output of MT, usually reflects the standard written language, which can be very different from how people actually speak, especially in the case of languages with large dialect and register variations.
> When I use MT, I have a simple rule, that I have to understand each word of a translation before I send it.
That sounds like an excellent rule. I will pass on that advice to educators I know who are trying to figure out how to guide their students on the use of MT.
Many thanks again.
If your aim is to make this technology more widely available and, as you claim, "give people the opportunity to access and share web content in their native language, and communicate with anyone, anywhere, regardless of their language preferences", then why make it so that the model essentially can't be used for anything useful? It doesn't really make any sense.
Legally even the use case which you're promoting on your frontpage - the Wikipedia Foundation's Content Translation - is illegal under the non-commercial license in certain jurisdictions! For example, see here: https://www.techdirt.com/2014/03/27/german-court-says-creati...
Even using it for research would be illegal as it's also not exactly "personal use".
How far off are my intuitions on this? What are the costs of adding a new language to a model like this? Is there a ballpark dollar amount per language?
Importantly, models can learn from other languages are similar. If we train separate models for each direction on small amounts of data, the performance is significantly worse than grouping languages in one large multilingual model.
The problem is that it increases the risk of monoculture to 100%. Without language barriers, cultural diversity is lost, not gained, since you have winner-take-all effects[0]. Instead of helping revive languages, it'll make American ideas, mores, morality (Puritanism), philosophies, and political values more dominant worldwide.
To be clear, this will increase economic opportunity, but will inevitably kill cultural diversity.
Is your team considering or studying this?
[0]: https://www.sampaxinos.com/mental-models/the-winner-takes-al... (or see Taleb's works)
>The affinity of languages allows one common model to be trained for their translation. That is, “under the hood” of the translator, the same neural network translates into Russian from Yakut, Tatar, Chuvash and other Turkic languages. This approach is called many-to-one, that is, "from many languages \u200b\u200binto one." This is a more versatile tool than the classic bilingual neural network. And most importantly, it is the many-to-one approach that makes it possible to use knowledge about the structure and vocabulary of the Turkic languages, learned on the rich material of Turkish or Tatar, to translate languages like Chuvash or Yakut, which are less “resource-rich”, but no less important for the cultural diversity of the planet.
>In order to create a unified model for translating Turkic languages, Yandex developed a synthetic common script. Any Turkic language is translated into it, so that, for example, the Tatar “dүrt” (“four”) written in Cyrillic becomes similar to the Turkish dört (“four”), not only from the point of view of a person, but also at the level of similarity of lines for a computer.
This way they added support for Turkic and Uralic languages which are very underrepresented on the Internet. But I don't know what the quality of their translation is: even though I live in a region where Mari is spoken (indigenous Uralic language) and my wife is Mari, none of us, sadly, speak the language.
[0] https://techno-yandex-ru.translate.goog/machine-translation/...
Regarding Mari: extremely interesting language, exciting to hear that you are from that region. We are interested in working on this one (likely in the "Hill Mari" variant), but currently do not support it.
So you have a language with some economic opportunity (a few million speakers in a fairly wealthy country) but no clearly defined written interface, and an ambivalent attitude of many speakers towards the very idea of writing the language.
As an aside, the question of how to think about language standardization is really complex. We wrote some thoughts in Appendix A of our paper: https://research.facebook.com/publications/no-language-left-...
If there are spoken, with enough budget, a library of voices could be recorded. I think you’d prefer that collection to be gathered and maintained by a non-profit rather than Meta.
The Facebook paper has some direct comparison to that work.
The reason for emphasis is in my opinion twofold: 1) Allowing these people to use the fancy language technology in their own language is good in and of itself. 2) Training neural networks on fewer resources is more difficult than using more resources and therefore a fun and interesting challenge.
Some numbers (but you can not exactly infer from them such accumulated numbers): https://en.wikipedia.org/wiki/List_of_languages_by_total_num...
Some more numbers from here: https://www.sciencedirect.com/science/article/pii/S016763931...
"96% of the world’s languages are spoken by only 4% of its people."
Although this statement is more about the tail from the approx 7000 languages.
Furthermore, Facebook considers the internet to consist of Facebook and Wikipedia (Zero).
I view this as just another extension of their Next Billion initiative, an effort to ensure that another billion people are monopolised by Facebook.
That's the payoff.
For instance, Icelandic is not supported by any modern smartphone platform which has lead to Icelandic natives communicating with each other in English and very little information is translated to Icelandic[1,2].
That being said, I am worried that having translations that are "too good" could also act to accelerate language death as the importance of keeping languages alive will seem less significant (to non-language-nerds) if we can translate works written in that language to any other language with very small datasets. Luckily I'm not convinced that AI models will be able to produce convincing and consistent translations for a long time -- languages are so different in so many ways that I can't see how adding more dimensions and parameters to a model would account for them.
[1]: https://youtu.be/qYlmFfsyLMo?t=141 [2]: https://www.nytimes.com/2017/04/22/world/europe/iceland-icel...
"Skills required: United Nations translators are required to have a perfect command of their main language and an excellent knowledge of, in most cases, two other official languages"
On one side they make their work easier as they can focus more on correcting the ai produced text and focus on author's meaning while eliminating lots of plumbing.
On the other hand they increased the amount of business because much more text is translated than at any other point in history, which requires validation in most business, legal and even personal contexts. Without ai translators those translations would've not happened in the first place.
Full story: https://abcnews.go.com/Business/wireStory/kill-facebook-fail...
These were submitted to test Facebook's systems, because there's a good reason not to trust their promises on this front. Facebook was used extensively to propagate hate speech in Myanmar during the crisis of 2017, with their moderation tools and hate speech detection system letting through a ton of hateful content with real-world consequences, in the course of an actual ethnic cleansing campaign.
Other references: "Facebook Admits It Was Used to Incite Violence in Myanmar" https://www.nytimes.com/2018/11/06/technology/myanmar-facebo... (2018)
"Violent hate speech continues to thrive on Facebook in Myanmar, AP report finds" https://www.cbsnews.com/news/myanmar-facebook-violent-hate-s... (9 months ago)
https://www.localizationlab.org/blog/2019/3/25/burmese-font-...
I see the mixture model is ~ 300 GB and was trained on 256 GPUs.
I assume distilled versions can easily be run on one GPU.
So, to clarify, does this mean that companies cannot use these models in the course of business, or is it more about selling the translation results directly?
also see section 3, table 1 in the paper: https://research.facebook.com/publications/no-language-left-...
I do admit I'm a bit bitter given another of the big silicon valley companies (Apple) claiming they specifically help the TCUs (Tribal Colleges and Universities) when I can find no one that knows about this help other than taking our money for product at the same price as other accredited educational institutions.
I have a feeling that if it was written in the original language it would go through, since many English names also have adjective noun original meanings like ‘beautiful flower’.
I have a feeling that if it was written in the original language it would go through, since many English names also have adjective noun original meanings like ‘beautiful flower’. The legal name is in English, so anyone expecting it to be written in the original language is expecting too much.
We shrug off all the little quirks of machine translated text because it usually gets the point across, and we recognize them as quirks because most of what we read was written by real people with no such quirks. But when most of what you read contain those quirks, I fear those will quickly become the standard way of writing and even speaking in those languages.
But there the rules are the same for everyone. This is not true in general; there are languages where men and women speak according to different rules.
Here's a selection from Empires of the Word:
> These works [written by women] are usually written in Emesal, 'the fine tongue', a separate dialect of Sumerian, well documented in scribal dictionaries. In dialogue works this dialect is used for the speech of goddesses. It differs from standard Sumerian, Emegir, 'the princely tongue', both in vocabulary (including the names of many gods) and also in pronunciation (consonants by and large being articulated farther forward in the mouth); it differs not at all in its grammar. For example, when the goddess Inanna is affecting to repel the advances of an importunate suitor, she cries:
> kuli Mulila šu bamu emeše daŋen amaŋu lulaše ta munaben amaŋu Gašangale lulaše ta munaben
> Friend of Enlil, let me free! Let me go to my house! What lie shall I tell my mother? What lie shall I tell my mother Ningal?
> Both Enlil and Ningal are, of course, gods. In Emegir this would have been (with the differences highlighted):
> kuli Enlila šu bamu eŋuše gaŋen amaŋu lulaše ana munaben amaŋu Ningale lulaše ana munaben
Point being, I'm not sure if language purity is more valuable than functionally allowing its people to interact with things they couldn't otherwise. Put another way, should we leave these people locked out of many online resources they can't read because we fear of corrupting their language? Give these people the option and let them decide. Language evolves over time anyway.
In real world instances (the proverbial 80%), it’s more often transforming a 0.4 (“don’t know much english”) into a 0.7. And the people who get away with near 0 knowledge will usually have no critical need for translation, or an access to other means (an actual translator, social help etc.) when really needed.
My mental image is grandmas reading online news, and machine translation would be a blessing and a curse. Or low grade school kids trying to look for some help on a topic, and a I’d wish they get more time with the original text to at least somewhat learn, than only getting the rough translation full of errors.
For interpersonal communication, people adjust, that’s what has been happening for centuries now.
I said nothing about purity, I said organic evolution, which this is an example of. If the actual speakers want to develop a pidgin, fine, I just think it should be a decision made by people and not models.
> These cookies are required to use Meta Products. They’re necessary for these sites to work as intended.
What cookies does Facebook "need" to serve a simple article?
But this is a shallow dismissal that doesn't add anything valuable to the discussion.
"Oh they made their _terrible_ (probably state of the art) machine translation _better_??! Those monsters!!"
(Edit: and speech-to-text models.)
Did the people at Meta think about the Signed Languages of the Deaf?
I didn't find a mention. Even Ctrl-F deaf didn't yield anything.
Understanding foreign culture is about reading automated translations of online comments into your native language. It has nothing to do with putting the effort into learning a language and understanding the nuances and current events and issues of the culture it embeds.
The ESL (English as a single language) speakers over at Facebook don't even need to understand foreign cultures, because they already know everyone in the world needs to spend their lives staring into the Metaverse. So grateful that they are working on the world's fattest pipeline for exporting Anglophone culture to every corner of the planet!