Very interesting. It's pretty good with song structure, formatting, themes, etc... The one major thing it seems to be missing is rhythm and rhyme.

I can imagine decorating the data with a pronunciation dictionary, so that you can coerce some better phonology from the model... There's a pretty good dataset from Carnegie Mellon here:

http://www.speech.cs.cmu.edu/cgi-bin/cmudict