Seconding it with: https://arxiv.org/abs/1610.03017 Fully Character-Level Neural Machine Translation without Explicit Segmentation.

In general, RNNs are especially fit for discrete sequences. And their continuous representation is actually an advantage (so they can see that two words are similar or analogous). BTW: see my draft on word2vec: http://p.migdal.pl/2016/12/30/why-do-word2vec-analogies-work...