MMA – Musical MIDI Accompaniment (2019)
mellowood.ca
mellowood.ca
We can do better today, see e.g. https://openai.com/blog/musenet/.
What do you mean by "better", and for whom? I've used a _very_ large amount of generative musical software and written a fair bit myself, and these are two different use cases.
MMA is a rather decent text markup that can allow a composer to sketch out an idea in terms of style, rhythms, structure, instrumentation, mood, chord inversions, decorations and passing - and then tweak that in a domain that is more or less one-to-one in effect, but still way above the level of editing individual MIDI notes.
Contrast that with tools that more or less wipe your arse for you, by churning out very interesting and intelligent "compositions", either via Markov, NN, clustering and reinforcement, genetic and evolutionary processes, tuned stochastic processes, generate and filter methods and so on... BTW, composition is one of the oldest application areas for "AI" around - I was building "Boltzmann and Perceptron Machines" in the 80s to do this on a 286 PC. What I'm saying is the application area is _well_ mapped over the past 50 years [1]
The latter methods great for getting inspiration and variations to select from - "play me something more like this in a rock style", but then so is listening to some music and chaning a keyword in an MMA file. For a moderately competent composer an intermediate-high level score, plus a good knowledge of the style libraries, is a far more desirable.
There is a market for "band-in-a-box" type products, where people want fast, fun backings to write lyrics to, or some kind of "in the style of" Karaoke machine for playing live at La Vista Del Sol Bar bar, but I can say from experience of working with a lot of _writing_ musicians that it's not what they're after, and it isn't clear if ML/AI can help much with that.
[1] see something like Curtis Roads CMT for a historical overview of paradigms.
More convincing/realistic musical performance. Imagine there was a "Turing test" for music composition and performance and the goal was to convince a trained musician that the performer behind the curtain was a true human being, not a robot.
> There is a market for "band-in-a-box"
Right, and the tool became much better since they use "real tracks", i.e. played by real musicians; I suppose in future these can be complemented by tracks generated by connectionist models like the ones used by MuseNet et al.
> see something like Curtis Roads CMT for a historical overview of paradigms.
See also Deep Learning Techniques for Music Generation -- A Survey: https://arxiv.org/abs/1709.01620
and Algorithmic Composition - Paradigms of Automated Music Generation: https://link.springer.com/book/10.1007/978-3-211-75540-2
You could even have "celebrity" performer profiles that learn Eric Clapton guitar performances the same way that synthesised voice style can be extracted from a speaker corpus.
A tool like that would be really fun and something I'd like to play with. It would take my composition and _perform_ the parts:
Drums: Ringo Starr, Solo Guitar: Steve Vai, Bass: Sting
In music it's quite usual though to separate the composition, arrangement and performance. TBH, what I found with _all_ ML/AI tools in the composition and orchestration/arrangement stages is badly lacking. Not because the "technology is bad" - massive leaps have been made with gradient descent and transformation, in scale, algorithms and hardware, but because the use paradigm doesn't appeal.
It's actually one of the biggest, painful learning experiences in my tech career, and I have taken it to heart - for example with the RJDJ startup we pioneered "reactive music" and had a ton of big artists on board, Chemical Brothers, Hans Zimmer, Massive Attack, .. and we made really credible, innovative music that pretty much defined a new genre and market that didn't exist. But it failed [1]. Why? Because people don't want their music that way. We made the classic mistake of thinking we knew what people wanted because _we_ thought it was cool (and it was, and investors poured money into it).
I used to excuse the fail by saying "it was ahead of its time". But as I get older I see that hubris alone was the sword we fell on, by thinking we understood how people want to relate to music.
There is such a concept as "fit" - I don't know what else to call it - that tech entrepreneurs often seem to miss.
That's why I say that a clunky old thing like MMA is actually a "better" tool, for me personally as a pro/am (once working) musician.
I hope that makes sense.
[1] AFAIK (I am out of the scene now) one of the people still doing this successfully is Matt from Coldcut with their "Ninja Jam" setup.
Well, that's mostly what classic musicology does (and some Jazz musicologists inspired by classics). At the latest when you only have a lead sheet and the "composer" expects the musicians to add all the rest, the interpretation of the piece is also an essential part of the musical content/"composition".
> the use paradigm doesn't appeal
It makes sense to first make some proofs of concept (as e.g. done with MuseNet) concerning compositional/performance quality and then adapt the technology to specific use cases. From my humble point of view there was no convincing result in that respect before e.g. BachBot or MuseNet - maybe with the exception of Cope's work, but his algorithms actually require a human selection and optimization step for useful results - it's not fully algorithmic.
> Why? Because people don't want their music that way.
Having the right ideas is just a small fraction of success; most is pure luck (from my humble experience/observations). But solving an important/interesting problem is not identical with pleasing a mass market. Concerning music composition and performance there are still big unsolved representation problems, i.e. how to model this n-dimensional problem so that all relevant patterns along each subset of dimensions can be properly described and related to other patterns/features. MuseNet has an interesting approach, but it's mostly a spin-off from linguistics and from my point of view still far from what we should have. There is no satisfactory solution since the beginning of music theory and especially since the beginning of the AI era. All we have are only rudimentary projections, makeshifts to have anything at all. That's why a "a clunky old thing like MMA" will never pass a trained musicians "Turing test".
Most good musicians will have distinct assessments of other musicians. "Good" here is a metric that involves a combination of time playing, time performing, time spent collaborating and possibly some innate abilities, should such a thing exist.
"I really enjoy playing with them, their sense of timing is so great"
"They are really good technically but their development ideas just don't go where I go"
etc. etc.
There's no reason to expect that any computationally based accompaniment system will do any better than the human ones: if it's actually good, it will have a "personality" that won't work for some people; if it doesn't have a "personality", it almost certainly isn't good.
There are many exceptionally talented musicians out there, measured by their technical abilities on their instruments. There are far, far fewer musicians that lots of other musicians want to play with.
In any case, studio session musicians have a very different set of skills to pop headliners. Arrangers have different skills to composers. Producers have some of all of the above, but are there to bridge the differing creative pulls of the artist and the record company and manage the project. Label executives and A&R set goals and make high level editorial choices.
When the public hears a finished track, it will have been worked on by tens or sometimes hundreds of people. Some of them - image and marketing consultants, PR people, social media managers - have no musical input of any kind, but still shape the subjective experience of the product.
As you say, all of the above are paid because they bring a personality and a unique creative flavour to a project. For a culturally significant and/or successful project, each flavour has to be distinctive but also has to blend successfully.
Music is always a composite product. Headline talent only gets you so far. The rest comes from creative tensions between individuals with different goals working in different domains.
For most styles - including classical - the public wants a narrative/image/hook fantasy that locates the product in a credible cultural space.
Generative music has almost none of this. Even if you create a perfectly polished pop single it will be culturally two dimensional because all of the surrounding context is missing. It will be flavourless - not just musically (most likely) but also culturally.
In fact there are spaces where this either doesn't matter much or (rarely) doesn't matter at all. But they're small, specialised, and not particularly interesting to work in.
Influential, interesting, highly paid projects demand strong and identifiable personal flavours. And they're far harder to synthesise than musically credible tracks/pieces.
Actually, I doubt this very much. I think there's a tendency to overestimate the "special" that actual humans bring to artistic creation and underestimate what contemporary machine learning (or I prefer to call it, overcooked cross-correlation analysis) can bring.
The problem is not so much that something generated using such techniques will be flavorless. It's that once it's good enough, it will have a distinct flavor, just like human artists, and that flavor won't sit well with everyone. That is to say, you can't solve this once: you have to solve it for every human aesthetic preference (within limits).
I can entirely believe in an automatic drummer that is "as good" as Steve Gadd, having been trained on the performances of said master percussionist.
However, if you want Gavin Harrison instead, you need a different instance.
And if you want something new, let's say 20% Vinnie Colaiuta, 30% Phil Collins and 40% Bill Bruford and 10% John Bonham, that will be very to accomplish through ML-style techniques. By contrast, Gavin Harrison could probably do that if you asked him nicely and paid him enough :)
Firstly the sound of all of those drummers was shaped by the people they worked with. This is most true of Collins and Bonham but the others were also very much shaped by the bands and producers they worked in.
There is no definitive "Phil Collins" style you can bottle. Phil Collins with Brand X sounds nothing like Phil Collins with early Genesis, or like Phil Collins with late Genesis, or Phil Collins on his solo albums.
It's easy to emulate the Phil Collins™ 80s gated snare, and of course in the 80s many people did. But even that was a collaborative effort and only happened when Collins did session work for Peter Gabriel who didn't want cymbals, so engineer High Padgham had to fill the gap. That sound is only a copy in a very superficial and trivial sense.
Secondly, Collins as a drummer is inseparable from Collins as a person. During the early Genesis years Collins was as much of a political influence as a creative one. He was relaxed, easy-going, and professional, in among a group of highly strung insecure public school boys who used to argue a lot. In all the interviews the other band members say that suddenly he made playing the music feel a lot lighter and easier, which very welcome but also hugely influential.
This is absolutely not something you can analyse easily by running his timing or drum choices through ML. Because the key part to both isn't the drum parts or the sounds, it's the creative ability to generate cultural influence.
So the idea that ML will give you 20% this and 50% that is a non-starter. Even if you could define what this and that are - very questionable - there's no guarantee you'll get something that has equivalent cultural reach, as opposed to just another drum part by a poor Collins (etc) imitator.
This turns out to be true of all the arts.
Culturally what people want from art isn't just technique and superficial style, it's a distillation and validation of relatable experience channelled through an extraordinary personality.
It's actually a relationship - albeit a very remote and idealised one.
You can only copy that if your AI actually has an extraordinary personality and is capable of personal experience which it can share. I don't think ML is going to be able to do that any time soon.
However, when someone wants to hire Gavin right now, it's almost always because they want "the Gavin sound" (whatever they conceive that to be). Although a musician's sound is to some extent always evolving, after 10-20 years of playing, it's fairly well formed, and evolving much more slowly and more subtly. Their musical and social history is "baked in", so to speak.
So, you train up your overcooked cross-correlation analysis system on the drum performances of Gavin Harrison, and just like GPT-3 "responding" to text prompts, I think there's a pretty good chance it will "respond" to your musical prompts in a style very similar to Mr. Harrison. However, it will sound nothing like Mr. Collins (except to whatever extent Mr. Harrison already did, which is ... not so much).
> So the idea that ML will give you 20% this and 50% that is a non-starter.
That was specifically what I said. Even so, I think there's some possible artistic intrigue that might come from some sort of "cross-breeding" between two systems trained on two different drum performance sets.
Good to come across someone else who at least knows that Phil Collins was an amazing drummer and not just the only other person besides McCartney & Jackson to sell a million records inside and outside a band. Still, Chester Thompson, seriously :)
This is a really great discussion. Can you elaborate a little about what you thought the public wanted, and what you found they actually want?
(I make some stochastic / algorithic music myself and am very interested in this.)
Sure I'll give it a shot, its over a decade ago now and I'm getting old at this. It was circa 2010 IIRC, and we had a a bunch of trailblazers, one Last.fm founder, a Second Life veteran, a top girl from Eve Online, me (procedural audio), and a vision for the delivery of "interactive" pop music - at the point of delivery, using loads of synthesis, but crucially using all the sensors to shape the music. So each "Reactive music" (coined by Michael Breidenbruecker) piece basically used the gyro, temperature, time of day, microphone, light levels and all that jazz... to create parameters for the music. We pioneered all kinds of stuff, like music that adapts to your running speed, driving, location (through GPS or other context awareness), sounds in the environment etcetera. I put a lot into the DSP and philosophy of "deferred form" as an artistic concept.
What we thought people wanted was basically "co-creation". We moved legal mountains over licensing stems, derivative works, sharing of UGC by audiences... and working with artists/producers to "capture" their workflow, reduce it to a rule set and stuff it into a smartphone (which they really didn't like for the same good reasons Aboriginal Aussies have around photography).
<Skip really complex psychological and philosophical stuff about identity, identification, art, attribution and social participation>
What people _actually_ want is "definitive form" (deferred form only works in dietetic contexts like video games =- thus procedural audio is a thing, whereas reactive music is not - although Kent Jolly and the Spore guys almost proved that wrong, and games still seems the natural home of algorithmic music).
Put it this way... everyone likes the idea of a "write your own story" game - games have infinite outcomes depending on what the user (player) does. Deferred form works there. But you can't make that work with film. I don't want a Star Wars where each time I watch it I get surprised: maybe Darth Vader and Luke _do_ rule galaxy as father and son, or Luke does a paternity test and it turns out "You're NOT my father!".
I want a Star Wars where everything in the universe is as I remember and like it, so I can watch it over and get satisfaction. I want my psychological "noo-noo" to play with. That's how people listen to music, and ultimately it's what they want - even if they say they want interactive smartphone music. This is why market research is a waste of money unless you grok the culture. You can ask them. You can build it. And they still don't come.
(BTW, I think this may be first and only post-mortem of RJDJ)
Listeners want definitive form most of the time, but what about "music for airports" or for elevators, etc. where a piece should be non-repetitive and yet last for as long as possible?
And what about creators / composers? Did they show any interest at the time?
Sure that's a thing. Waaay back I built a system to go in Harrods department store and in other supermarkets, basically SBCs of similar grunt to Raspberry Pi's now, running Csound. I made generative scores to create sound effects and ambient music all day long. That company, The Sound Agency, is still going AFAIK.
> And what about creators / composers? Did they show any interest at the time?
Well of course. But here's where I have to bite my lip, because to be honest they don't always.. well artists. Eno obviously gets it. Guy coined the "Generative music" thing in so many ways... but Eno actually releases recordings. He selects the good stuff and puts it out as definitive form. I've always argued that's a different art form from what we were doing in the Shoreditch arts hacker scene back in 2005-2012. We were releasing stuff as code, so songs and albums in Supercollider or Pure Data - very much inspired by the German demo scene where it was common to release music as a .exe and each time you run it you get something a little different.