MusicLM: Generating music from text

I love that there's a whole section for accordion examples. I think the accordion rap has a bad word in it. Accordion techno works surprisingly well.

They really need to bump up to 48 KHz so all the music doesn't sound like it's being played over a telephone. A factor of two in the cost shouldn't be prohibitive. So much of the audio generation stuff I've seen has fatal flaws like this baked into the dataset and/or training process that ensure the output can't sound good even in theory, it's kinda frustrating.

It's also frustrating that nobody AFAIK has trained a music model on an actually large dataset. We're training large language models on a significant fraction of the text of the whole internet! Where are the audio models trained on a significant fraction of all recorded music? This one was trained (in part) on a dataset of 280k hours of music, if I read the paper correctly. I don't think that comes anywhere close to being a significant fraction of all recorded music.

Epic soundtrack using orchestral instruments is somewhat interesting as electronic music but to me everything else is pretty bad.

There is a million Industrial techno sounds, repetitive, hypnotic rhythms IDM tracks that absolutely no one listens to anymore. Most the founders of the whole genre have moved on from lack of interest.

We have been able to do really good generative music in Reaktor for the last 20 years. Much better than anything on this page but no one cares.

People in general want to hear the same slight variation in music over and over as background noise.

The people that are saying how great all this is will be bored with it in 2 weeks or less. Pure technological kitsch.

Holy hell those are insanely good. Best generated music I've heard so far.

I was born too soon to explore the stars, born too late to be a 17th century pirate... but born just in time to explore these incredible neural networks come to life.

The vocals (on second page of table) are really interesting. If you walked into a club, you might not notice they are nonsense, but it's just good enough BS to be convincing.

MusicCaps dataset: https://www.kaggle.com/datasets/googleai/musiccaps

Totally fails on "swing" and "relaxing jazz".

This should replace the OP link

dope, can it queue up weird al? I am curious how generative would work at coming up with parody

May I recommend to learn how to use a tracker software? It is very fun to play with!

We have a AI platform finding music from video/images/game screen recording. And support musicians at the same time. Check it outhttps://avmapping.co/

I hope you also consider getting some music from an actual producer!

https://samplab.com is doing something like that.

Yes! And not just notes or chords or time signatures, but real analysis of how the piece works.

I believe parts of this model could be adapted to accomplish this. The "mulan" network generates embeddings from music and text so that they "match" each other, i.e. audio segments and text captions that go well together would be mapped to the same embedding. At run-time, they use embeddings from the text to generate the music. So conceivably the opposite could be done, using the "mulan" embeddings from audio only to generate text, by replacing the "soundstream" decoder with a text LLM. That said, there's probably simpler methods that could generate decent results, and I'd be surprised if there isn't already some out there. It's worth noting that along with this work they released a dataset of music with captions that could also help training new models for this task.

MusCaps: Generating Captions for Music Audio: https://arxiv.org/abs/2104.11984

Working on the same dream: https://parture.org

I think you’re missing the point. Music is an experience, not a checkbox or text book needing summarized.

> especially the conditioning on humming and whistling ... but to bad they use very common melodies

100%. Now if you add the detailed "Painting Caption Conditioning" to the mix, you can create a melody in the style of… an image, which, in a way, is kind of a controlled artificial synaesthesia [1].

[1] https://en.wikipedia.org/wiki/Synesthesia

Possibly just anecdotal evidence, but it seems like the pool of good music composers is much much smaller than the pool of good visual artists. Maybe that is an indicator of how "hard" each field is.

> which is 1-dimensional and has probably much less internal modelling to care about

There is an ocean between “I like that sound” and a final, produced piece of recorded music. Much of that ocean being ineffable.

Trying to reduce it to a set of parameters around the final waveform and you’ve missed the entire point.

> but it will suffice for, say, putting a music background for your startup cheap marketing ad.

We need less of that—not more. It’s like a climate disaster of the soul.

In my opinion, good music is multidimensional.

Just like a 3D model is really only 2D on a screen, music can encompass so much more than what is heard on the surface level with no imagination.

It will be as good as the music it's trained on, no better and probably no worse.

I understand what you're saying, although it could be argued that at least for some types of image tasks one would prefer something like an SVG output with layers to make it easier to edit.

For music, I think it's partly an academic question of "can we do it" rather than trying to maximize immediate practical usefulness. There's already quite a bit of work on symbolic music generation (mostly MIDI), a lot of it quite competent, especially in more constrained domains like NES chiptunes or classical piano, so a full text-to-audio pipeline probably seemed a more interesting research problem.

And for a lot of use cases, where people might truly not care too much about tweaking the output to their liking, the generated audio might be good enough; the examples were pretty plausible to my ear, if somewhat lo-fi sounding (probably because it's operating at 24kHz, compared to the more standard 44-48kHz).

In the future a more hybrid approach probably makes sense for at least some applications, where MIDI is generated along with some way of specifying the timbre for each instrument (hopefully something better than general MIDI, though even that would be fun; not sure if it's been done). I'm sure that in the near we'll see a lot more work in the DAW and plugin space to have these kind of things built-in, but in a way that they can be edited by the user.

You can absolutely "overpaint" sound, as sound can indeed be represented as an image and be extended in frequency of in time. In fact this is exactly what Riffusion is doing.

“For a song you need midi”

Pretty sure the Beatles never handed George Martin any midi files. What’s the symbolic representation that captures the tone of every bend in a Hendrix solo? Did Daft Punk go back and grab the raw master stems of the old vinyl recordings they used to assemble their tracks?

Music producers have been astonishingly creative given inputs in a vast range of formats. Sheet music and midi are one tool, but ultimately it’s about combining sounds in the mix isn’t it?

That's quite a harsh critique that doesn't seem warranted. There are quite a lot of different examples with varying quality. Granted, it's likely that these are to some extent cherry picked, but it's unlikely that these are rare and that most output sounds like incoherent noise, as you seem to insinuate.

Their current demos seem worse than riffusion results I get on average. Music gen is hard because music is inherently a composed of many many different instruments, each with unique sound and function. Simply training end-to-end will almost always end up badly.

Well... no all releases have to be toys for your average twitter user to play with. Most of them are just academic stuff to show team progress.

For this specific paper, probably never. From the final section:

"""We acknowledge the risk of potential misappropriation of creative content associated to the use-case. In accordance with responsible model development practices, we conducted a thorough study of memorization, adapting and extending a methodology used in the context of text-based LLMs, focusing on the semantic modeling stage. We found that only a tiny fraction of examples was memorized exactly, while for 1% of the examples we could identify an approximate match. We strongly emphasize the need for more future work in tackling these risks associated to music generation — we have no plans to release models at this point."""

Knowing Google, they will somehow never released it by invoking a "think of the children", like every single one of their ML papers.

> Anyone want to share their experiments with MusicLM, feel free to join

How can we experiment with MusicLM?

The paper’s last sentence says, “We have no plans to release models at this point.”

> if a note is slightly off, you instantly hear it

I don't know about this. I know professional musicians and they say that they make little mistakes all the time, but part of being a professional is being able to pretend those mistakes didn't exist because the audience doesn't know what it's supposed to sound like.

The fact that mistakes in music are so obvious comes down the the fact that underneath music is a lot of structure and pattern. That there can be a ‘wrong’ note implies that there is redundancy in the signal and predicting the right next sound is precisely the kind of thing these sorts of generative AIs excel at.