First off - used this code for training the models: http://karpathy.github.io/2015/05/21/rnn-effectiveness/
Very very easy to setup and train; highly recommend playing around with your own training data (just a text file!)
This project's code: github.com/shariq/burgundy
Styled and deployed the website about a year ago at a hackathon; it then used a nice wordlist with hand picked words. (repo/wordserver/old_burgundy_words.txt)
Few days ago: got the server to start training a bunch of models (~200), with randomized parameters, using the original wordlist as the training data. (repo/rnn/rnn.py:forever)
Yesterday: woke up at 3 AM after my sleep schedule rolled around, started exploring the output of models trained to different numbers of epochs and run at different temperatures. Subjectively looked at the outputs, decided some model/epoch/temperature tuples were horrible, got rid of those. Wrote a few different scoring functions (just using intuition for what kinds of bad outputs seemed to be commonly occurring) to score the model/epoch/temperature tuples. Got the top ~10 scoring tuples from each scoring function, plus added some additional interesting ones along the way, and then used a pronunciation scoring function (repo/rnn/pronounce.py) to select the top 5 of all of these. Funny enough, the top 5 tuples all used different models and a varying range of temperatures (i.e, not the same model from different epochs, and picking the right temperature significantly improved how well the model performed) (repo/rnn/explore.py)
Since the models would still occasionally output words which were completely unpronounceable, I put some code on top of the models which would generate a bunch of words then discard the bottom 1/3rd of unpronouncable words. A significant portion of generated words from these models also started with a "c" or "b" for some reason: gave those a high chance of being discarded. Short words were also uninteresting, and extremely long words would occasionally show up: added probabilistic filters for length. Finally, initialization time of LuaJIT is very high, so I had the server keep a pool of words which gets reseeded as it runs out. (repo/rnn/rnnserver.py)
If you want to train your own word generator and you need some pointers, would love to help: @shariq