I think the interesting part would be example based audio synthesis. Could you replace a synthesizer with a neural network which, when fed examples, would allow you to generate sounds / explore some latent space between the examples.

For example an approach similar to https://gauthamzz.github.io/2017/09/23/AudioStyleTransfer/ but then using the methods described in the PixelNN paper.