Do NLP Beyond English
ruder.io
ruder.io
The situation for those who, like myself, work in applied linguistics is not actually so dire... At least when working with the languages that have a reasonable amount of training data. Decent enough treebanks exist for lemmatisation, POS-tagging, and dependency parsing for dozens of languages [0]. Fast tools such as spaCy (15 languages) [1] and udpipe (40+ languages) [2] are freely available and work well for most applied tasks. There are even decent word embeddings available trained on various versions of Wikipedia [3].
Of course, some of the issue is that these very tasks are biased towards the specific structures of Indo-European languages. However, for getting work done (building sentiment classifiers, document clustering, NER, ect.), currently available tools make it possible to work with a large proportion of the currently available data.
There is still a lot of work to be done own regards to computational research on non-English languages, but a lot of the problem right now is recognition of applied work by the top NLP conferences rather than a complete lack of quality work currently being done.
[0] https://universaldependencies.org
[2] https://ufal.mff.cuni.cz/udpipe/models
[3] https://github.com/facebookresearch/fastText/blob/master/doc...
It helps of course that English is the de facto lingua franca of most of the technologically developed world (if not the rest of the world also). If another language was the lingua franca, current approaches would not work so well. The article makes a point about how we assume that pre-trained embeddings encode all relevant information about a language when this may not be the case for arbitrary languages. The article also makes a point about the morphological simplicity of English which is of course a great asset for a lingua franca, and also simplifies the language modelling task. Imagine if the lingua franca was, e.g. ancient Greek (as it was, in ancient times, all around the Mediterrannean) or something with similar morphological intricacies (and Greek is still an Indo-European language; I just don't know much about, e.g. Finno-Ugric to give an example about them).
However, you can still incorporate language-specific knowledge if you want to get the best performance for that one particular language. This is mainly true for academic research. If you are developing a rule-based or keyword-based application (which is still often the case outside of academia), it can actually help to be fluent to get the rules right.
Providing context may help, but still translation to English strips important pieces of information.
I think in general we pack a concept in a word and lose some information this way, so when you want to be precise with what you are saying you have to bring your definitions with you. Essentially with translation you take a concept and "pack" it in a word, then look for an equivalent packing in different language, then unpack. Naturally, this process is prone to losing information.
Reading translated books from Polish to Spanish or Russian to Spanish conveys a lot more information, than reading the same book in their English translation.
It's like every subtle nuance is lost in English.
Depending on the form of the thing you are translating it can be simply impossible to translate properly (like poetry).
I'm currently reading English translation of "Black Obelisk" which I believe is written in German and it isn't any worse than Russian translation (my native language) to me.
In any case, what was originally asserted is that English is somehow worse than other languages as a "transitional" translation language for words or simple phrases, so I argued with that specific idea. Translating literary works is a subject of it's own and where the quality is much harder to measure.
Funny that the upstream commenter essentially praises Spanish as superior to English, Spanish being the language I dismissed as less expressive than English when I was a noob.
A bad Netflix or literary Spanish translation, for example, is full of frustrating ambiguity ('who is the antecedent of "su" here? "Bajó"? Who bajó?? At least in English we have to use "he" or "she"!'). And, with experience, you realize that native Spanish writers will keep things disambiguated, it's just crappy translation shortcuts that don't. And trying to compare translations is only an exercise in comparing translators.
Though you said it better than I did.
This isn't an issue for any considerably long fragment of text, it will be properly translated due to context analysis. Still if the text would be analyzed using German in the first place, this would become less of a problem.
Linguistic complexity is hard to measure, but it's not hard to show that at least morphologically, English is undercomplex compared to many other languages.
This doesn't necessarily mean that English is more ambiguous, though. Unlike German, English typically has very rigid word order, so in the context of a sentence, you'll know if a specific word is a noun or a verb.
The problem here is that many NLP models inadequately capture syntactic structure.
> Unlike German, English typically has very rigid word order, so in the context of a sentence, you'll know if a specific word is a noun or a verb.
So you say you are able to guess from the word order what part of speech a particular word is. But with German you hardly need all this guessing.
If you compare two marginal examples: - English "time flies like an arrow" - German "Wenn Fliegen fliegen hinter Fliegen..."
you'll find out the English one has way more possible interpretations.
Not really. It's not about guessing: in English, the part of speech really is mostly determined by its syntactic structure.
> If you compare two marginal examples: - English "time flies like an arrow" - German "Wenn Fliegen fliegen hinter Fliegen..."
Not sure what you're trying to say here. The English example is ambiguous, yes (and only strictly grammaticaly; semantically the meaning is clear, unless you're using it in the phrase "time flies like an arrow, fruit flies like a banana", which is meant as a linguistic joke). It's also very easy to come up with examples of phrases or sentences that are ambiguous in German, or in any language for that matter. Here are some fun examples:
"Er liest das Buch seiner Schwester vor" (could either mean "he's reading the book to his sister" or "he's reading his sister's book to someone")
"der weiße Schimmel" ("white mould", or "white horse")
"wilde Tiere jagen" ("to hunt wild animals", or "wild animals are hunting")
and don't even get me started on the ambiguity of compound words or phrases with a genitive, where there are often tons of potential interpretations depending on the intended relationship between head and dependent noun.
And also the German example you gave (fully: "Wenn Fliegen hinter Fliegen fliegen, fliegen Fliegen Fliegen nach", or "if flies fly behind flies, flies fly after flies") is a) another joke sentence nobody uses in practice, and b) is exactly a case where you can only distinguish the part of speech (and the grammatical case) of a word from the syntactic structure and not from its morphology, something you claimed doesn't happen in German, but here it clearly does.
Look, you may make a case that it's easier for English sentences to be ambiguous than for some other languages, but I would need to see some good data before I believed that claim, because it's just not something that is immediately obvious.
> is a) another joke sentence nobody uses in practice, and b) is exactly a case where you can only distinguish the part of speech (and the grammatical case) of a word from the syntactic structure and not from its morphology, something you claimed doesn't happen in German, but here it clearly does.
I didn't make such a strong claim. All I wanted to say in German syntactic ambiguities are much less of a problem than in English. I've brought two anecdotal evidences to let you compare possible ambiguities in both of them, these two are indeed nothing but jokes.
But let's take a closer look at them once again.
a) "Time flies like an arrow": the word "time" can be 1) a noun 2) an adjective 3) a verb in declarative form 4) a verb in imperative form. This gives us a factor of 4 on the very first word of the sentence.
b) "Wenn Fliegen hinter Fliegen fliegen" - ambiguitity exists just between "fliegen" as a verb and "Fliegen" as a plural noun, thus the "ambiguity factor" of the word "f/Fliegen" is just 2.
> but I would need to see some good data before I believed that claim, because it's just not something that is immediately obvious.
Fair enough.
This is a plausible description of how humans perform translation, but it does not apply to machine translation, because we have no good way to represent the meaning of a word other than with the word itself. Consequently machine translation systems can't distinguish between different meanings of the same word and instead try to produce a correct translation by relying on frequency-based heuristics: faced with two likely translations of a word, a system will try to determine the context of the word (in terms of its collocation with other words) and then assign to the word the meaning it has in the context that happens to be the most common according to its training dataset. Clearly, that is like "flying blind"; sometimes it will work, sometimes it will fail and there's no way to predict beforehand which.
The comment above gave the "spring" example, my routine example is asking Google Translate to translate Greek "χελιδόνι" (the bird, swallow) to French and getting "avaler" (the verb, to swallow) instead of the correct "hirondelle", again because translation goes from Greek to French via English, introducing ambiguity about the intended meaning of "swallow" that does not exist in either Greek or French. Note that this doesn't happen when the word "χελιδόνι" is used in a sentence (e.g. "ένα το χελιδόνι" translates to "un l'hirondelle", which is ungrammatical and nonsensical but at least gets the right noun), but it's a good test to show that Google Translate is really incapable of recognising the meaning of words and so cannot use such information to make translations. Note that the same goes for machine translation in general, i.e. Google Translate is a state of the art system.
Now the main reason (at least, I think it is) it's handled poorly is that Dutch has an infinite number of words ( * ) and order in compositions matters a lot. This conflicts with the typical precomputations that try to reduce the number of words etc.
( * ) take any 2 nouns A & B. you can concatenate them into a new noun AB or a new noun BA (with completely different meaning). German has this concept too.
They aren't that common in Dutch anyway unless they have evolved into a more or less standalone noun. Dutch nor English people really think of a flamethrower as a thrower of flames, it's just another noun.
This is also how it works in English, except the concatenation may be written with a space as "A B". If anything, writing it as "AB" should make search more accurate, since you're less likely to get results with "BA" than getting results with "B A" when searching for "A B".
But while it's technically true that English also has compounds, it just writed them apart, it is also the case that German (don't know about Dutch) uses compounds, and especially long compounds with more than two nouns, a lot more than English, where you might more often use a whole sentence.
I wonder if commenters have considered that Research is useful to understand the present and more importantly prepare for the future. While English being useful língua Franca today seems to back decision to just focus on English research the point is: will this English uptake continue at same rate in future? Isn't there some kind of bubble effect where here in hn many think that (as non English native speakers) we can all engage so we think English is the only useful thing for us to understand?
What about all possible unknown Mandarin NLP research (which I think many agree) may be useful for non-mandarin speakers?
Also, isn't one of the points of Information Technology to enable/create value in a long tail of experiences? Focusing on a single attribute of a substantial part of the world seems counter productive and shortsighted...
It's like we are all in some sort of Reinforcement Learning algorithm and unifying the language portion would significantly restrict the exploration phase, the one that moves world forward. There are many concepts unique to each language, different emphasis on different things, different emotional response to different sentences, leading to unique ways of exploring the universe, and that all would be lost.
There is a Sapir-Whorfian assumption here that it is the language that enslaves the mind, rather than the collective minds that are constantly shaping, morphing and bending the language. To me, the prospect of cultures having to communicate with each other via such a broken mechanism as translation is far more repugnant than a linguistically unified superculture within which ideas flow unimpeded.
The more language users there are, the more capable of expression the language will become; so no, I am not particularly worried about getting stuck in a local maximum.
The world is ultra connected now. It's fine for people to be bilingual and continue to communicate and respect their local history and culture. There's also a premium on being able to communicate wherever you go and with anyone you might meet. That's where English is already filling a dominant niche.
The concern that some countries have with children learning English (when it's not their native language), is that their children might also grow up being heavily influenced by American/British culture (as they'll grow up watching English TV shows, reading English books, etc). That's a legitimate concern, where it's difficult to decouple the language from the associated culture. That said, I think there's a value in being able to communicate with everyone else and those countries could do other things to either limit screen time or increase the amount of historical and cultural content they give to their kids (this seems mostly like a problem for people who just want to sit their kids in front of a laptop and hope for the best - active parenting and teaching of local culture seems effective).
You say this as if there were consensus around that, but that's not true. The very strong, original formulation of Sapir-Whorf probably doesn't hold much water, but you'll still find enough linguists claiming that language influences thought nontrivially. But of course, you'll also find linguists arguing the opposite.
You don't need Sapir-Whorf to see the point. Even if language is purely a tool, only relying on one language narrows cultural expression. Korean for example contains honorifics baked into the grammar of the language to express social status, this is not trivially doable in English, at least in any version that could be considered unified with what is called English today.
I am not sure I understand what you mean.
Sure, there are languages whose grammars require speakers to choose a more polite or a less polite form of a word; but I doubt this presents an insurmountable obstacle to the other culture for expressing the attitude of politeness, or the concept of hierarchy. Being speaker of a language that has two forms for the word "you" — a polite, plural one, and a familiar, singular one (I suspect that this grammatical behavior was imported from French in around 18th century; the distinction is similar to what Middle English had with you vs thou), I am sure that our concept of politeness is not markedly different from cultures whose languages do not have this distinction in second-person pronoun. The distinction cannot be trivially translated into modern English, true, but I don't think it matters a whole lot.
Besides, if a culture really starts to feel a need for expressing a certain concept, they will find a way for doing that with the language they have.
https://en.wikipedia.org/wiki/Honorifics_(linguistics)
Korean has seven different levels of speech depending on the level of casualness one engages in and the Japanese language has features to raise or lower the status of the addresse as well encoding different feelings in the language.
There is no analog for this in English, and also not really in Western culture. There's a very distinct notion of politeness in several Asian cultures that has no corolarry in Western societies because we don't tie it formally into social status.
There's also a lot of other issues. A lot of major religious traditions rely on revealed scripture whose meaning is only precisely accessible through a particular language. In Islamic theology this would be Arabic and for Jews it is Hebrew of course. It would be an extreme cultural loss for billions of people to lose this access to their traditions.
I don't see it as a problem in the slightest, and not just because I personally think religious traditions are better left behind. No-one these days speaks classical Latin or ancient Greek; and yet we have some knowledge about the ancient tradition through commentaries and translations (regardless of how imprecise those translations are). We do not experience this as a tragedy or as an extreme cultural loss, despite us not being able to read those texts. Nor is anyone bemoaning the inaccessibility of Beowulf and the inadequacy of existing translations. And if someone is especially interested in the topic — well, there are always scholars who are free to learn those languages and to study the originals.
I do believe that we miss out on a lot of nuances by not being able to read Latin or Greek. But I also accept that, pragmatically, we can't all be learning these languages anymore, so yeah, I'm not one of those grumpy Latin teachers. That doesn't mean it's not still a loss.
Imagine a world of 7billion English speakers: there would obviously be a huge range of accents and standards. In fact, compared to other languages, English is much less standardized and I'd argue that the premise that there's a single "English" for NLP to focus on.
So, I'd rather NLP scientists find fulfilling work in exploring existing languages diversity.
While standardization certainly lends convenience and efficiency, diversity leads to more robustness and effective exploration — instead of everything being caught up in the same bubble. The web as a platform might become unmanageably “hot” (in a McLuhan-esque sense) if that were to happen; we’d just have to find other arbitrary ways of differentiating ourselves to form more convivial communities.
I don’t mind if English becomes the common interchange format (lingua franca, ha) but I hope people continue to be multilingual and conserving and growing the cultures of other tongues.
English is already something of a standard for international communication I think, at least in some domains. It looks like a successor to Latin in many regards.
Nonetheless I don't think English can function as an universal language because it is specialized in some regards. It is better for thinking about and capturing some things and ideas, but not for everything. Other languages, it seems to me, are optimized better for certain use cases.
Everything of course cross-pollinates and, thanks to that, flexibility of different languages increases. But some fundamental rules must remain in place to maintain coherence of a language. If you change the basic grammar rules, you get a different one. In this way achieving universality within a single lanugage is either impossible or very difficult. Hard to tell if lanugages are evolving towards convergence into a single universal superlanguage or if the optimum is specialization.
Programming languages seem to follow similar patterns.
Curious as to what you had in mind (assuming it was broader than Eskimos and snow!)?
Care to back up this claim with examples? To me, it sounds like you're advocating for linguistic relativism [1], which doesn't really hold up to our modern understanding of language and thought.
In general, it's not fair to say that linguistic relativism doesn't hold up to modern understanding. The debate is still on-going.
[1] https://anthrosource.onlinelibrary.wiley.com/doi/abs/10.1525...
As I said in another comment, I think there is a feedback loop between language and mind, so IMO strong relativism or determinism creates a false dichotomy. As Tainnor mentioned, the article you linked provides some examples that suggest the weak form of relativism or however you want to call it exists. At the same time it agrees that determinism is false.
To my amateur eye this just shows that a feedback loop exists. Anecdata to back it up would include writers, scholars, philosophers, many of them polyglots, like Conrad, Watts, Nabokov, Huxley, and many others who seem to express a similar sentiment. In a polyglot context this hypothesis seems to arise often, see for example this interview:
http://writeparagraphs.blogspot.com/2014/02/a-stroll-with-al...
> I can draw polar opposite philosophical lessons from my experience. On the one hand, languages are only fascinating (or frustrating, depending on your point of view) as long as they are foreign – once they become familiar, that is, once they are simply used as a vehicle for communication, then they all do the same thing. So, in this sense, no matter how different they might seem, they are all the same. Obviously, this is true of us as human beings as well.
> On the other hand, each language has certain unique characteristics, words, and concepts that do not exist in other languages. Furthermore, there is an enormous variety of ways to express ideas and thoughts. Many of these ways of expressing what is in the brain seem absolutely essential to those who use them, but in point of fact they are not, as other languages do without them entirely.
> Finally, as far as culture is concerned, all languages are created equal and all languages do carry the culture of their speakers if by culture we mean simply the ways and traditions of a group of people.
or this article:
https://www.theguardian.com/education/2013/sep/05/multilingu...
> Each language resonates with me in a distinct way, bringing out a different part of my character. Russian makes me more melancholic because of its minor tone. In French, I am super pensive. Brazilian Portuguese is a very flirtatious and sweet language.
> But there is more to it than that though – each language has its own way of expressing thoughts and ideas, so you get a real insight into diverse thinking. Language carries the culture of the country that uses it and when you internalise it, it becomes a part of you too.
etc.
My personal experience with Polish and English also aligns with this. Switching between these languages influences how I build thoughts. Sometimes I seem to make mistakes in Polish because my mind is more used to thinking about a certain area in English and the other way around as well.
For expressing some thoughts I find Polish more suitable with its ubiquitous prefixes and suffixes and flexible word order. For other thoughts English seems better with its ubiquitous homonyms and simplicity. Sometimes it's easier to make leaps of understanding and connection between different concepts in one, sometimes in the other.
These are the two languages that I think and speak daily with something close to 50/50 ratio, perhaps more English, even though it is my second language.
Other languages that I've been exposed to also seem to show me different ways of thinking. For example German feels more rigid then either English or Polish but also more lego-like. Chinese or Japanese seem to be quite different still, with their peculiar subtlety. Arabic, written, with its single case letters (I was pretty surprised when I learned that), simpler in some areas than other languages, more subtle in others. Even the fact that different languages use different writing directions is itself kind of mind-bending when you first learn and think about it.
All the best to them :-) I think having models in nations that actually use a language as their native tongue is far better than choosing a language for which there is no living model (such as Latin or Esperanto).
But then, why don't you also add Britain, Australia, Canada or South Africa to your list?
edit: I personally think Esperanto would be a good choice as an International Language. I believe it's much easier to learn than English.
Moreover, it is well known to sociolinguists that political and economic power is something that makes a language spoken by that nation more prestigious and more desirable to attain, and, therefore, contributes to the spread of that language. So, for the purposes of my dream, I can only wish that the US remains as influential as it is (and doesn't switch to Spanish in the process) :-)
I guess my point is that I am not sure that, say, having other countries where English is the primary language (as listed in the previous comment) contributes to the power of the US. Likewise, I am not sure that the further spread of English as an international language will by itself contribute significantly to the power of the US.
> I personally think Esperanto would be a good choice as an International Language
Ugh, not a fan. English is a vibrant, living language that already is being used by huge swathes of people for various purposes; Esperanto is not.
The opinion expressed in your comment, that we should adjust ourselves to our technology, rather than improving our technology to suit our needs, is not entirely new. For example, Andrew Ng has said that rather than trying to make self-driving cars capable of dealing with an unexpected situation such as a pedestrian on a pogo stick, we should make sure that people don't engage in such unexpected behaviour that self-driving cars can't handle [1]. Obviously, this is a recipe not just for suppressing much of the diversity of human behaviour with unpredictable results for our own cognitive capacity, it's also backwards and antithetical to the motive of improving technology to make it capable of new applications.
___________________
[1] Rodney Brooks, Bothersome Bystanders and Self Driving Cars:
https://rodneybrooks.com/bothersome-bystanders-and-self-driv...
Languages are generally not unique and this is how humans are able to deal with sparse data as well. So you can use a language model that is trained on text across all languages and then the model will infer common structures to languages with sparse data. XLM did this and seems to perform fairly well although I don't know how sparse you can go before it breaks down.
> Doing well with few data is thus an ideal setting to test the limitations of current models—and evaluation on low-resource languages constitutes arguably its most impactful real-world application.
This is what ML and Stats has done for the last X decades and it has limitations due to the need to constrain the problem with human generated rules/knowledge/axioms. The new large scale language models skip the human rules step and are thus able to learn the long tail of language structure. I don't see how you can have it both ways as the unconstrained problem is infeasible to learn on small data as there isn't enough to generalize from.
I mostly agree with you, but it does seem like current models require more data to learn language than baby humans.
This would suggest there is a middle ground between old fashioned data starved learning, and current overfed models.
Contrast that with language models which do nothing besides watching what people are writing, it potentially points to a reason who the amount of data is not comparable.
No matter how close you might get to "the language learning algorithm", if all you're feeding it is text, it's not going to learn the same thing that kids learn. The data is simply not there.
- The grammar for standard Mandarin is much simpler than English. No verb conjugations or strange tense rules ("go" -> "went")!
- Tense and even voice are explicit and additive - i.e. adding "呢" or "吧" to a sentence.
- Written, it seems less ambiguous. i.e. "duck" or "grave" have different meanings in English based on context, but that seems much much rarer in Chinese - two words spoken the same way but with different meanings usually have different written forms.
But it's possible that focusing more on Mandarin would still lead to improvements in other areas, because the two languages are otherwise quite different (I wonder how well speech recognition works for tonal languages, for example?).
[1] Possibly not without reason: there is some indication that languages spoken by a large number of speakers tend to become structurally simpler, while very small isolated languages can often develop surprising complexity.
This is an interesting observation. Training on large datasets tends to be framed as a strength, e.g. we have recently seen articles praising OpenAI's GPT-3 for being, well, big. The truth is that model size (and the assorted dataset size) is a bug, not a feature. It is the result of a dearth of research on approaches with low sample complexity. Or in other words, it's the result of consistently picking the low-hanging fruit and calling sour grapes on any domain for which there isn't sufficiently large data ("who cares about Swahilli?").
>> In contrast, most current methods break down when applied to the data-scarce conditions that are common for most of the world's languages. Even recent advances in pre-training language models that dramatically reduce the sample complexity for downstream tasks (Peters et al., 2018; Howard and Ruder, 2018; Devlin et al., 2019; Clark et al., 2020) require massive amounts of clean, unlabelled data, which is not available for most of the world's languages (Artetxe et al., 2020). Doing well with few data is thus an ideal setting to test the limitations of current models—and evaluation on low-resource languages constitutes arguably its most impactful real-world application.
And this is interesting to read in the context of the recent manic hype about GPT-3, itself described in a paper titled "Language models are few-shot learners", hilariously attempting to sweep under the carpet the amount of data and compute required to get to the point where a language model can do "few" shot learning.
In general, the excitement about the successes that have come from training large, deep neural nework models with very large datasets has only served to avoid posing the obvious question about such data-hungry approaches: what do we do when there isn't a lot of data? Human language with all its minutely fragmented diversity, turns out to be just that kind of domain.
Granted, our ml isn't even close to human intelligence yet. But why is the assumption made that humans are "trained" with very little data? We've collected millions of years of data to accomplish what we have.
>> Also, it's not like any human is just born fully capable of inventing an entire language. It seemingly takes entire societies to pull this off.
This is a complicated matter and as I'm not a linguist I don't know what theories there even exist. However, I think a human child is perfectly capable of inventing an entire language. Small groups of children are certainly capable of doing that in the early stages of their lives. For example, see Creole languages [1] which are the languages invented by the children of people who only have a pidgin as a common language. A pidgin is not a fully developed language and pidgins are used e.g. for communication between migrant communities with different origins and who do not have a common language, while a Creole is a fully developed natural language that is invented by the migrants' children, who have their parents' share pidgin as a maternal language. The children start with their parents' pidgin and develop it into a fully formed natural language with all the bells and whistles. It's quite a thing to read about really, have a look at the wikipedia article I link above it's very interesting.
Also, as far as I understand it there's a lot that we don't know about how the first human languages were created. From some short readings, the current understanding is that at some point, humans must have become capable of producing natural language whereas before they weren't. The way things usually work the turning point for this was most likely some random mutation that affected a very few individuals, possibly no more than two. So the first natural language was probably invented not by an entire society but by a couple of people, probably a pair of siblings. The idea is that the ability to use natural language gave those very few individuals an important advantage that ensured its propagation through generations until our time.
In any case that all goes to invention of a new language. Learning an existing language is quite another matter. Children who grow up to learn their maternal language don't have access to "millions of years of data". They certainly don't have access to a few billion examples of utterances in their maternal language, like the datasets that language models are trained on. That is to say, kids learn whatever language is their maternal language very, very quickly as they grow up and from very few examples. And they're capable of learning any of the few thousand natural languages, beyond English. For me at least this lends credence to an idea of an innate bias to learn a certain type of language, a type broad enough to cover every human language. To put a name on this idea, that's through and through Noam Chomsky's idea of an innate "Universal Grammar". See wikipedia about criticisms of UG and linguistic nativism in general [2].
____________
[1] https://en.wikipedia.org/wiki/Creole_language
[2] https://en.wikipedia.org/wiki/Universal_grammar#Criticisms
I'm not yet sure whether that is a good or a bad thing for non-English-as-a-first-languege regions.
Surveillance capitalism relies on getting ever more info about you, and exploiting it or selling it on to all that want to exploit it. NLP is one vector to extract that info from raw data.
I thought I'd give an example of this for readers who don't speak many languages other than English. My native language is Greek, which is an Indo-European language, but it has some morphological intricacies that make it more complicated to learn and use (as a second language and from what I'm told of course) than English.
For example, in Greek, nouns have a gender [1]: masculine, feminine or neuter. For instance, "the dog" is "ο σκύλος" ("the male dog") while "the cat" is "η γάτα" (the female cat). However, grammmatical gender is not necessarily fixed so one can also say "η σκύλα" ("the female dog") or "ο γάτος" ("the male cat"). So there is a word-root ("σκυλ-" for dog, "γατ-" for cat) that is then modified by a termination, typically -ος, -η -ο for the nominative of each gender. The terminations change depending on the declention, for example the genetive for "σκύλος" is "σκύλου" and the generative for "σκύλα" is "σκύλας", etc. The termination also changes to denote number, singular or plural (ancient Greek also used to have a dual number, used to refer to pairs of nouns). Terminations vary depending on number according to gender and declention. So for example the plural of "σκύλος" is "σκύλοι", genetive "σκύλων" and the plural of "σκύλα" is "σκύλες", genetive "σκυλών".
Compare this with English where a dog is a dog is a dog, there is only one plural form, "dogs" and there is only one genetive form "dog's" or "dogs'" for each number. And a female dog is either a circumlocution, "female dog" or an entirely new word, "bitch", with its own simple set of transformations for number and genitive "bitches" and "bitch's" or "bitches'". Basically, changing meaning in English can be represented by adding a few words to a vocabulary- but changing meaning in Greek requires manipulating words at the structural level.
There are probably a few counter-examples to the above but my understanding is that this is how it works for the most part. And this, in particular the abilty of English to represent more meanings with more words, might go some way to explain why approaches that work best with large datasets tend to be favoured in English NLP.
Edit: not an NLP expert or a linguist so corrections welcome, of course!
______________
[1] Other Indo-European languages also have genders but the interesting thing is that those genders don't always match, between languages. This is an endless source of confusion for foreign language learners. For example I speak French from an early age and I only realised much later in life that Greek and French do not always use the same gender for the same nouns, even though I'd been using the right genders in my speech. For example, both "dog" and "cat" are masculine in French: "le chien" and "le chat". But while I always used "le chat" correctly, I always thought of it in my mind as "η γάτα", the feminine Greek noun. Human language is weird!
Difference mainly lies in data abundance, where the distribution is severly skewed.
NLP in other languages can help sell better ads in those languages.
You all can follow that chain of thought down to it's corporate conclusion =)
1. Provide materials in their own language
2. Consume materials in their own language
3. Actively and exclusively contribute to the world in English
it is already a statement about the necessity of their language: it isn't. Why should non speakers of a certain language invest money and/or time on a language its own people don't even invest in? If its own speakers kill the language (actively or passively) who are we to say they shouldn't?
Because rational choices at an individual level do not necessarily align with rational choices at a social collective level. "Meditations on Moloch" is a good read on this topic. You can easily see how collectively the language diversity is positive, but individually it kinda makes sense to bet for the biggest language instead. And to be fair, there's a lot of people investing in their own languages, just not enough, or not with enough power.
A continuing lack of technological inclusion will not only exacerbate the language divide but it may also drive speakers of unsupported languages and dialects to high-resource languages with better technological support, further endangering such language varieties. To ensure that non-English language speakers are not left behind and at the same time to offset the existing imbalance, to lower language and literacy barriers, we need to apply our models to non-English languages.
Have you considered what this means, concretely, for a speaker of an endangered language? Suppose I want to "invest" into my language and
1. Provide materials:
- Written? I need some kind of keyboard support. In the worst case, this might entail getting new characters added to Unicode. Or I could just use paper, but then I lose out on all the benefits of digital text editing.
- Spoken/Signed? In some ways, that's easier, since I can just record myself. But if I want closed captions or some kind of transcript, I can't make use of automatic tools, I have to transcribe it completely manually.
2. Consume materials:
- But how do I find them? What if searching for words in my language mostly turns up results in other languages because the search engine assumes I misspelled my query?
- It might happen that the text I'm trying to read uses characters that were considered "too rare" to be included in the default font on my device. On most versions of Android, it's not possible to install a new font to fix that. I'm not sure about other mobile operating systems.
Those are problems that don't require speakers of the language to do something differently (such as just using English instead...) but instead need technical solutions that can very well be implemented by someone who doesn't speak the language. I don't speak an endangered language myself, but I've written code to support dozens of languages I don't speak, because someone who did speak the language asked for a particular change that would make life easier for them.