Comment by zozbot234 | Hacker News Reader

Comment by zozbot234 | Hacker News Reader

Parent Full thread

This comment should not be disregarded so easily. The reason why deep sequence learning has the best results in generating complex, highly contrapuntal music (it's more like noodling or improvisation than an actual compositional process, but it is generally compelling at its best) is precisely because of the loosely grammar-like structure mentioned in OP. The algorithmic operations they play with are not very well defined but the background theory is sound, and closely reflects what music theorists and composers in general have written about the subject in the 500 years or more it has been seriously studied.

As for deep learning models which create good contrapuntal music, see e.g. 'Biaxial RNN' https://github.com/danieldjohnson/biaxial-rnn-music-composit... by Daniel D. Johnson, who is now at Google Brain but wrote this as an independent(!) researcher. (Note that the existing code requires Python 2.x It would be interesting to forward-port it so it can work with Python 3.x and a maintained version of Theano. Replicating the model using Tensorflow would also be quite worthwhile.)

If you're interested in Bach's work specifically, the "BachBot" and "DeepBach" projects are also interesting but less accessible.

Example output for all of these models can be found on the Internet, just look around for it. The proprietary system AIVA is also worth mentioning because even though it's so proprietary and secretive, the compelling and "serendipitous" music it manages to come up with is a tell-tale sign that it's actually doing well-founded deep learning stuff behind the scenes, much like the aforementioned open systems. Note that much of the released output has been orchestrated (AFAICT) manually by humans, but at some point I was able to find some piano-format reductions that are most likely very close to what the AI actually created, somewhere on the official site.

I was inspired by your comment and took a look at Daniel's work.

Forked it and spent the day upgrade to Aesara (the maintained Theano fork) and python3.

Haven't fully trained/tested the model, but feel free to take a look: https://github.com/kpister/biaxial-rnn-music-composition

Whoa, thanks! You could try issuing a pull request once the work is complete. Most likely the author will accept it, or else it can linger as an "open" request on github and others might find it.

Also, the old version could use a special incantation:

  theano.config.experimental.unpickle_gpu_on_cpu = True

to run CPU-based inference on a GPU-trained model, even without a supported hardware GPU. It would be nice to check how this stuff works with the newer packages, and document it properly. Most likely nothing much has changed.

Deep learning needs a lot if training data. There’s not enough MIDI encoded music to train something like a GPT-3 model to generate MIDI sequences. A better way is to train on and generate raw audio, see OpenAI JukeBox. Unfortunately it’s extremely compute intensive (even compared to GPT-3), so it will probably be a few more years until they (or some other big player) releases JukeBox-2.

The amount of data you need depends on the model architecture you're using. A generic model like GPT-3 is neither here nor there, but something specifically intended for music can make do with very little data.

What do you mean “neither here nor there”?

The structure of something like GPT-3 is far too weak and general to achieve good results for something as structurally complex as music. It's designed to generate text - and then mostly natural language text. Music is very different, as OP hints in the linked post.

I thought you wanted to use deep learning. In DL transformers are the best we got currently. Both JukeBox and MuseNet use transformers. What makes you think they are not up to the task?

"Transformers" is a general technique, comparable to "LSTM" or "attention". The details of just how much prior information about the domain you reflect in the model architecture are just as relevant, and it's not clear just how well MuseNet addresses this compared to the earlier work I mentioned above. As for JukeBox, it's trying to solve audio generation at the raw samples level, which is quite literally a complexity increase of several orders of magnitude.

A transformer architecture does not change whether you use word tokens, musical chord encodings, audio waveform samples, image patches, or video frames. You can change a number of layers or attention heads, but the architecture remains the same - it does not care what is the nature of sequences it encodes/decodes. That's kind of the point - to have a universal algorithm. Human neocortex does not care which sensory receptors the signals are coming from, there's evidence that it processes all of them in the same manner.

Any prior information about the domain can be encoded in a way you prepare the input samples, and this would be equally useful and relevant regardless of the DL architecture you decide to use (RNNs, autoencoders, GANs, transformers).

Using raw audio samples has two advantages: First, you provide all available training data to a model, and trust that it will learn to extract what it needs from it. Second, unlike any other music representation formats, you have huge amounts of training data. So essentially all you need is a huge transformer and many thousands of GPUs to train it on, using all mp3s you can get your hands on, and you can reasonably expect a similar level of quality as the quality of text generated by GPT-3. What do you think?

> Any prior information about the domain can be encoded in a way you prepare the input samples

Adding features to provide further information about the input data can be helpful, but not nearly as much as ensuring that strong priors are reflected in the actual model architecture, whenever relevant to the domain. Johnson's paper about biaxial RNN provides one example of how to do this in the context of music.

Reply on news.ycombinator.com