That depends on the order of the model you use, and the complexity of the phonology. For an over-simplified example, consider that English allows both 'st' and 'ts' clusters; a 2nd-order Markov model might thus end up producing something like stststupid, which is clearly not a valid word.

A 3rd-order model will of course do much better, but will still fail to a greater or lesser extent depending on language. And the higher the order of the model you use, the more it will just spit out the same stuff it was trained on and the less it will behave creatively, so there is a tradeoff there.

I agree it might not make much difference here, but there are good reasons for investigating more complex kinds of models for language generation, and RNNs are an interesting choice.

False; try making something similar with a Markov model trained on just 800 words: the RNNs have somehow learned what it means for a word to be pronounceable, and rarely output words which are difficult to pronounce.