But there's no reason the models are english specific...
data --> system --> model
It would be unremarkable if the same system would have to be retrained from scratch to work with a different language, but I take it that you mean more than that. One guess is that the training data is annotated with grammatical information, in which case I would wonder if this is just a shortcut, or whether it solves a fundamental problem for such systems. Another guess is that the training set includes disambiguation, but that would seem to render meaningless the results on Winograd schema. (Update: I withdraw this last point, as presumably the Winograd-schema questions themselves are not disambiguated. Disambiguation of the training corpus would be a quite significant language-specific accommodation, if that is what is happening.)
It really is that unremarkable, despite the GPs insinuations to the contrary.
In any case, I don't remember anyone ever naming their n-gram models or HMMs etc. The reason for this of course is that training a language model with a deep neural network architecture is anything but unremarkable, not least because the costs in data, human-hours and compute are beyond the reach of most entities other than large corporate teams. If training a new model from scratch were truly "unremarkable" we'd all have our own.
If, on the other hand, a given architecture could not be retrained to perform well on a different language, or only on a group of related languages, that might be indicative of a fundamental limitation in how far it could be developed, depending on what sort of accommodations have to be made for new languages.
Basically retraining the same system on a different dataset is a bit like entering the same river in two points in time. Yes, in theory it's "the same" system and only the dataset chnges. In practice, it takes so much architectural tweaking and fine-tuning that it's a different system and you've done almost all the work from scratch. This also is part and parcel of why so much fanfare surrounds the publication of a new model. Because it's really, really hard to train a new model on a new dataset (assuming one wants to get state of the art results).
Actual NLP practitioners are not having a hype-fest every time someone spends a shitton to train a transformer LM on a new dataset. It's definitely cool, but I wouldn't cal it remarkable. What was cool about GPT-3 was the demonstration of the effectiveness of the "few-shot"/"conditioning" approach. I didn't find it surprising, but it was cool to push it to the next level.
> In any case, I don't remember anyone ever naming their n-gram models or HMMs etc
That's because they are literally different model architectures. The 3-gram model is the same everywhere, it doesn't depend on dataset. A 3-gram "trained" on Russian would still be a 3-gram. Similarly, BERT trained on a German corpus would still be BERT, it wouldn't need a "new catchy name." That's my point.
As a side remark, you are vastly overestimating the amount of dataset specific tuning that you need - you can often just use the exact same hyperparams and run it on a new dataset - there's been some recent literature showing that a lot of these architectures are actually quite resilient to different datasets and hyperparameter choices and arrive at similar local minima. Sure, you might be off by 1-2% from SOTA accuracy (let's say on a classification task) but in production systems you're generally not shooting for the SOTA that was released in the last 2 months but rather something close-ish to those results.
Honestly, quite confused this is the hill you want to die on.
>> That's because they are literally different model architectures. The 3-gram model is the same everywhere, it doesn't depend on dataset. A 3-gram "trained" on Russian would still be a 3-gram. Similarly, BERT trained on a German corpus would still be BERT, it wouldn't need a "new catchy name." That's my point.
N-gram models are not "the same everywhere". An n-gram model is a set of probabilities assigned to, well, n-grams. Different sets of n-grams, or probabilities trained from different datasets would be- different. The n-grams would be different and the probabilities would be different and the structure of the model would be different. A Russian n-gram model would represent the probabilities of Russian n-grams, an English one would reprsent the probabilities of English n-grams and so on. The type of model would be the same, an n-gram model; but the model itself would be different. Otherwise, why bother training more models?
Same goes for a model trained with a transformer architecture. BERT is an English language model trained with a transformer architecture and representing the probabilities of tokens in an English corpus. A model trained with a transformer architecture and representing the probabilities of tokens in a German corpus would be, again, a different model. You could also call it "BERT" but that would eventually become very confusing.
>> Honestly, quite confused this is the hill you want to die on.
I'm not dying on any hill. Could you please not do that?
I don't really think it's super worth quibbling over, since I think we're generally on the same page. It's worth noting that BERT trained on different corpora is generally still called BERT (you can look at the huggingface/transformers library, for instance).
I'm sorry that my message came across as more adversarial than intended.
Thanks. I enjoyed our conversation.
Standard Chinese can be spoken in Cantonese, but while understandable, it is never done, in contrast to Mandarin.
In recent years written online conversations (like blogs, Facebook posts, instant messages) by Cantonese speakers are increasingly written in Cantonese instead, and this is slowly spreading to more "formal" media such as books.
Source: I'm a native Cantonese speaker.
They're still a lot closer than English and Spanish or even French and Spanish (https://www.ezglot.com/most-similar-languages.php).
E.g. I bought this book: ngóh(S) maáihjó(V) nìbún syù(O).
Contrast with Japanese: watashi wa(S) kono hon o(O) kaimashita(V).
Differences from Mandarin Grammar are listed here: https://en.wikipedia.org/wiki/Cantonese_grammar#Differences_...
Classical Chinese is also SVO: https://en.wikipedia.org/wiki/Classical_Chinese_grammar
This applies to all languages BTW. There are only very, very few NLP systems that actually work, and they are all in a setting which tolerates a fairly high error rate (e.g. ranking).
This has a lot more to do with smaller Russian corpora than anything to do with peculiarities of the Russian language.
> This applies to all languages BTW. There are only very, very few NLP systems that actually work, and they are all in a setting which tolerates a fairly high error rate (e.g. ranking).
I've trained NLP models used on over 15 different languages used in production
I'm taking bets.