Unsupervised Machine Translation Using Monolingual Corpora Only [pdf]

Neat! If I'm reading this right (smarter people please correct me if I'm wrong), the process they used is:

1- Train a system that translates language A sentences into a representation space, and can translate back from that space.

2- Train a second system that does the same, but with language B, onto the same representation space.

3- Train an adversarial system that tries to look at the representation space and identify which language the sentence came from, retraining the language translation systems to try to fool that recognizer. Retrain the models to try to not be recognized by this third system.

The best way to 'hide' from the recognizer is to have a very similar distribution, to make it use similar points in the representation space for the same concepts.

Brilliant stuff.

It seems like there's a hard limit on how well this approach can work, defined by the extent to which the distribution of words in texts uniquely identifies their meaning without reference to any external semantics.

With monolingual corpora and no semantic grounding, all you can say is that certain words (or letters, morphemes, or whatever) covary in certain ways that are similar to the ways in which other words covary in the other language. It's super interesting that this is enough to identify some translation pairs across languages.

This is pretty interesting, and something I've played around with too (although not to the extent they have - I was playing with aligning word embedding and using them for cross lingual tasks).

Google released a paper[1] doing zero-shot translation between unseen pairs. That relied on a shared representation which they called "interlingua", and that seems quite similar to this

[1] https://research.googleblog.com/2016/11/zero-shot-translatio...

This is pretty cool! But I would've liked to see some eval on smaller datasets (or just a graph of BLEU vs training set size), since the main use for something like this is where you don't have parallel data, in which case the monolingual resources are likely to be much smaller too. A few years back I was trying different methods to create bilingual dictionary word-pair candidates (to be post-edited by a linguist – checking is faster than writing) for Saami languages, and tried aligning pure word vectors, but the existing corpora were too small to get good vectors, so I gave up that avenue :(

Another neat approach is https://www.isi.edu/natural-language/mt/RLILdecipher.pdf , specifically designed for low-resource languages that happen to have a higher-resource related language (e.g. Haitian Creole–French). They train a source language character-based "cipher model" (a weighted finite state transducer that turns source characters/character pairs into target characters/character pairs), to maximise probability over a target language language model.

What an awesome idea. I wonder how well this technique would work for different data types. Text and audio for example.

Feels reminiscent also of CycleGAN (https://github.com/junyanz/CycleGAN)

At first I thought this was really insightful now it seems like just redefinition of meaning.

An analogy is an experiment where a scientist says "I can prove these two people have ESP and can pick the same #". He sits them in two separate rooms and has her first pick a number. He goes to the second room and has him pick a number. 17? No higher. 24? No higher. 52? No lower. 43? Yes exactly!!

"See, these two people both came up with the same number without any communication with each other. ESP!"

If they want to show this language learning is powerful, demonstrate that once trained it can now be applied to a third language without any new "adversarial feedback".

Or it reminds me of the fake "we can communicate faster than light via quantum entanglement" claims. But then the caveat comes that they can only figure out what the value of the bit of info they communicated was causally.