So Jukebox can actually be broken down into 2 components / 3 steps: A) compression (take 44khz audio and compress into tokens using 3 VQVAE layers) B) generation (take the compressed tokens and use that on a transformer to generate sequences of those tokens) C) de-compresss (reverse the VQVAE) -> this is BY FAR the longest/most computationally expensive step

Diffusion currently wont really help with (B) or even (A)... but there is a lot of new experiments going on with (3)... where the decompression using stable-diffusion is providing very interesting results using far less compute... check out JuicyJukebox notebooks (link to come... cant access github colab from my work computer).

Yes, I believe it had been done with Google’s Wavenet for example where normally it’s trained to generates speech conditioned on textual input. For fun they trained it on piano music and gave it no conditioning and it improvised piano pieces.

Also see OpenAI’s jukebox.

Jukebox is using a similar scheme: https://openai.com/blog/jukebox/#approach