Suppose I have languages X, Y, and Z. My machine currently knows how to translate between X->Y and X->Z. The goal is to turn Y into Z without direct training. The process would be to translate Y into X and X into Z.. effectively Y->Z.
This isn't really transfer learning as much as it is logical induction...Or am I missing something?
It is one of the oldest ideas in machine translation. In the late 1950s, Richens proposed an MT system that used an intermediate representation called an "interlingua." For each language, the system has two components, one that maps a natural language onto the interlingua, and a second that converts the interlingua back into a natural language. Use different source and destination pairs and--bam! you've got a multi-way translator.
However, this only works if a) the interlingua is rich enough to capture the semantics of the to-be-translated text and b) the conversions between natural languages and interlingua also preserves those semantics. In practice, neither has worked pretty well.
In the 1960s, the interlingua was typically hand-crafted and rules were either hand written, or later, induced, to convert natural languages to/from it. Since people have been working on this for 60 years, you can probably image how well that works[].
The clever bit here is that you don't really need to do that. If you have enough data, the LSTMs can learn it on their own (see Figure 2, for example, where semantically-related sentences from multiple languages end up in the same neighborhoods).
[] Actually, somewhat better than you'd think. It certainly wouldn't have done Pushkin any justice, but it was surprisingly decent on weather reports (etc).
Just trying to re-think in terms of a Russian sentence instead of an English sentence has also proven difficult, and I'd be curious how you can automate it on NN's well. My colleagues (russian natives) are all able to spot auto-translated messages from me nearly instantly since it messes up pretty rudimentary grammar rules or just doesn't phrase the sentence like you would in native Russian.
However, you could almost read the paper as "We can translate something from a natural language into a high-dimensional representation, then turn that high-dimensional representation back into (another, possibly different) natural language. "
The two tasks are surprisingly interchangable. I once worked on a project where we used a statistical MT approach to "translate" between image features and captions--and I don't think we were the only ones trying such things.
In a pleasing bit of symmetry, the attentional network used here looks like it was initially developed for image captioning.
So that is new.
P.S. note the hand-waviness when I use the word "emergent"