It's getting to be within spitting distance of being completely adequate (though not excellent) for translating long-form articles, which is amazing considering where things stood ten years ago.
The neural networks are great at faking it, and they'll make up text that looks completely reasonable all day long. If that means inventing subjects and choosing random verb tenses, that's what they'll do. If you pick closely-related languages, you might be able to get a near perfect translation just by putting each word through a dictionary, and the neural networks can disambiguate things and smooth over the problems in the output by looking at local context. If you pick distantly-related languages, you might routinely need context from paragraphs away, or context which isn't present in the text at all but relies on some kind of human experience.
low resource translation is definitely still bad, you're right.
> near perfect translation just by putting each word through a dictionary, and the neural networks can disambiguate things and smooth over the problems in the output
Funnily enough, this is not a terrible explanation of how the current sort of approaches to low-resource MT work.
> low resource translation is definitely still bad, you're right.
No, you've misunderstood. Languages that are closely related to each other are fairly easy to translate between. Languages that aren't related are not, regardless of whether you have a large corpus to work with. English - Mandarin is not an example of "low resource translation".
The size of the corpus makes a huge difference. I don't see how can both be true that a. there is this massive failure of translation engines on these language pairs,
b. human translators were unable to differentiate between human translated chinese-english sentences and machine translated sentences.
It's not incredible - but it's also not awful or unusable.
If your use case is (1) being vocally excited about machine translation, or (2) having fun in your spare time, then machine translation is good enough for you.
If you have a genuine need to know what something in a foreign language says, machine translation is unusable.
This isn't really a good usability profile.
In a sense, I feel that this is an even worse situation, because the word-salad signal that it isn't an accurate-meaning translation isn't as strong.
Maybe this was true 4-6 years ago, but no longer.
I'm curious what the domain is and what languages you're talking about, it's hard for me to evaluate your claims otherwise. I would buy it for English -> Tibetan for instance.
My observation of the failure rate (if you define failure as "meaning has been lost") is well above 80% with Google Translate for these languages and this type of text.
It's literally unusable because you have no confidence that the translation has not completely reversed or garbled the meaning of the text, and the failure mode is particularly unpleasant because you may be unaware that it has failed to translate correctly, since it produces text that appears to have meaning - just the wrong meaning.
Your comment about Chinese feeling like English written in Kanji is really interesting to me, perhaps it's a case of different perspectives - I tend to focus on grammar when learning a language, and so Chinese feels very different to English from my perspective.
But Asian languages are nevertheless related in more mysterious ways. Compare the intro to https://en.wikipedia.org/wiki/Mainland_Southeast_Asia_lingui... :
> The Mainland Southeast Asia linguistic area is a sprachbund including languages of the Sino-Tibetan, Hmong–Mien (or Miao–Yao), Kra–Dai, Austronesian and Austroasiatic families spoken in an area stretching from Thailand to China. Neighbouring languages across these families, though presumed unrelated, often have similar typological features, which are believed to have spread by diffusion.
In a European context, this is similar to the unusual closeness between the "unrelated" English and French, known to be due to intensive contact between the languages post-William-the-Conqueror. (In a world context, English and French are closely related regardless.)
It's not "literally unusable." Even for language pairs with poor translation, it considerably saves on manual translation time if you can have a translator look over sets of already translated (but lower-quality) sentences and correct those with errors.
Plus, there are actual studies out there comparing human translations to machine translations in a double-blinded evaluation by translators and the gap is not as high as your comment suggests.
For chinese-english (the other direction), bilingual evaluators have had difficulty differentiating between the machine translation and the human translation.
Having attempted to use CN->EN machine translation many times, I find this very surprising. Have you got a reference I can have a look at?
https://news.ycombinator.com/item?id=24302564
Bonus clip: https://www.youtube.com/watch?v=0E4aYgKmIko
-> Human evaluation does not agree that PBSMT is more accurate than NMT. Sure, the fluency/"sounds good" gains from NMT are much larger than the accuracy gains, but evaluations I've seen put NMT ahead on both metrics.
Before, I could skim outputs, and get a fair idea of which areas were poorly translated by where it was garbage. Now, I get the impression that (like with GPT or style transfer) something always comes out, whether it's related (or even necessarily has the right negation parity) or not.
And it is also worth noting (as you've sort of mentioned) that appearance of fluency can bias translators who are instructed to just evaluate accuracy, or to evaluate accuracy and fluency separately. There is ongoing research into that problem.
“- Milk tea or regular tea? -Yes!” Yes is the proper response in some languages. You can either change the language or your mindset.
And it’s not like these things are one offs, all languages are basically a bunch of quirks.
No doubt it’s the way of the future but I’d wait for the researchers to improve translation a lot.