Check out Char2Wav (recent) and SampleRNN (the RNN-based audio synthesis architecture). The related work section of the Deep Voice paper mention a bunch of related papers that are relevant!