I put your question in talktotransformer.com and got this response:

  I know people have been having trouble adapting these kinds of
  generative techniques to text. Do you know of anyone making
  interesting progress there?
  
  
  It is difficult to make progress in the field of generative methods
  with text alone - it takes effort and creativity to get a generative
  system working. A big part of our research focuses on generating
  sequences which correspond to handwritten data, and to improving on
  that method we have developed generative techniques which allow us to
  generate a large range of novel sequences. Our work is still small,
  but we are not stopping there, in fact our next major research project
  is to generate novel sequences for novel languages.
  
  
  What I see is that we are still in an early stage when it comes to
  the technology used in generative methods to create new words, but I
  suspect this is due to a combination of factors. First and foremost is
  the fact that the techniques we've developed to generate novel
  sequences are highly specialized in a particular kind of context -
  we are not going to create random numbers or sequences because that
  just doesn't work. Generating a word, for example, uses very specific
  computational principles and can only be done if you are aware of the
  context in which it is being generated (or "determined" as the
  linguists would say). Even so, the general principle has been around
  so long, that one could quite easily create several different methods
  to create.