The template here looks like it would go
"<script src="jquery.js"></script> <script src="sha1.js"></script> <script src="links.js"></script>
<script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-4209567-5', 'auto');
ga('send', 'pageview');
...[until generation limit, ~1600 characters; note the absence of any valid closing tags]
The low quality of the text might reflect the demands of simultaneously modeling text+HTML+CSS in a too-small RNN or insufficient training (training a high quality char-rnn/torch-rnn to convergence on the entire Geocities corpus, or even just a few hundred megabytes, could take weeks).
He wasn't training on his own computer so aside from being much too small (for modeling at least 3 languages, text+HTML+CSS, I would start with 1k neurons and go up from there), it probably wasn't trained to convergence either. The 50MB size is maybe a bit on the small size but probably not the root cause of the low text quality.
It's not clear to me that NN are the best way to get interesting Franken-pages. Maybe a Markov chain instead?
I suppose that NNs have some sort of limit to how many layers they handle, but I thought they tended to learn nesting fairly well, unlike markov chains?
I could be wrong about this.
It also has to do with quality of input. The texts on geocities pages aren't exactly pinnacles of clear and legible language. Same with the HTML.