A.I. Duet: A piano that responds to you | Hacker News Reader

A.I. Duet: A piano that responds to you

aiexperiments.withgoogle.com

262 pointsdsr1270 comments

I like the concept and the framing of the demo is really cool, but the actual music generated in response to me doing basic things like playing a blues riff or a pentatonic scale was usually just a repeat of some fragment of what I was playing. It felt like something that could be built using a basic heuristic...

Agreed, it seems to either play a note you've played or pick a random interval. The interval and timing perhaps aren't truly random but it makes no difference to the listener's ear. Doesn't appear to have any concept of, or the ability to learn, scales, rhythm, or melody (or chords). Neat idea, but akin to playing piano with a 5-year-old (perhaps not as fun).

Of course that may change as people could be expanding its training through their interactions with it.

Yeah, had the same impression

It seems it almost gets the concept of tone, but barely

And for some reason it worked on Firefox, but not Chrome (it doesn't even load for me)

Could just be a limitation of the training resources available.

I don't think there's a practical limitation on training resources with respect to Music. Maybe the authors didn't get into any of the rubric, but it's certainly out there. Gobs of it.

By resources, I meant training time, CPU/GPU resources, not availability of data.

At Google? Ha.

It almost felt random at times.

Yeah either the examples from the video are cherry-picked, or something's wrong with the deployment, because what I get is way worse than what's in the video.

I really don't like this new trend of throwing Deep Learning at anything without really checking whether a simple model (e.g. markov chains in this case) would give equal results. Because using a simple model wouldn't be "cool" or newsworthy...

When you've described the entire new vision of your company as being "doing things with deep learning", you aren't going to bother seeing if the simpler solution works. Just isn't good for business.

But then a competitor comes along, doing the same thing with vastly less complex model and capable of offering cheaper/faster service because of that.

Here's a relevant excerpt from research I did on college into this. FWIW, my research was specifically into genetic algorithms and generating similar musical styles as the input music. It didn't perform particularly well but had some surprises.

From my research, a Markov model could potentially be viable for this demo in particular. However, it wouldn't be viable for longer generation of music where you're trying to produce an actual song. The biggest problem being that it doesn't have the concept of melody in it.

--- research after here ---

A Markov model utilized by Herfort and Rehberger [17] uses the immediate previous note in order to determine the next note against a set of probabilities. This is a fairly simple algorithm as the set of rules is not large, but it is extremely short-sighted by nature. This “first order” Markov chain is not capable of determining a probability based on a large sequence of pitches. To counteract this, “N-order” Markov chain models can be determined to look more deeply into the past, but the complexity of the rule set increases exponentially with added depth. Further, every aspect of music that is to be modeled would require its own Markov model. For instance, there would need to be a model for pitch, duration, etc. This leads to a potentially huge Markov model rule set where all chains are independent of each other. Thus, Markov chains are good at composing musical melodies on the micro level (such as a measure) but fail to produce quality music at the macro level.

[17] Rehberger, Leopold. “Composing music using AI algorithms.” GRIN Publishing GmbH. 2004. <http://www.grin.com/en/e-book/108991/composing-music-using-a....

Source: https://github.com/sb8244/GeneticMusicComposition

Just to complete your post, one problem I always have with music generation using "first-order" Markov Chains is that they get really confused by children's music. For example, here's the start of "twinkle twinkle, little star":

CCGG AAG- FFEE DDC-

So the tranistion matrix contails rules like: P(C -> C) = 50% P(A -> A) = 50% P(F -> F) = 50% P(E -> E) = 50%

Which makes huge loops of the same note quite common. This makes for some really uncanny music.

even going a simple jingle bells loop (I know, I haven't studied much) get's a pretty much nonsensical repeat response.

on a side note I'd like to have an option to put ai on mute and have the duet run after a session, since it's extremely disorienting for a noob to play with random noise on the feedback loop

Me: plays a couple of minor chords

AI: long pause; holds down one random key for several seconds; boops another random key "Nailed it."

After spending a few minutes playing, I did see evidence of it occasionally tracking my chords (as long as they were very common - e.g. c major, a minor) and tempo (but only matching tempo, as in vaguely matching how fast I was playing, not actually playing in time with me). I played an extended chord sequence and it made one or two nice contributions, but mostly it was just random noise.

I wonder if this is due to lack of training data? Maybe it needs an actual data set of people specifically trading musical phrases with each other, rather than just a set of general music.

It's an amazing idea, and I can actually see it being quite interesting for things like musical phrase creation - supply the first half of a phrase and see what the machine comes up with for the second half, and tweak as appropriate. But currently it's a bit of a case of over promising and under delivering.

Definitely a work in progress. Was like having a goddamn cat following my fingers around the piano. No sense of rhythm or tonality whatsoever. Like, it objectively made whatever I was playing sound worse.

It may "respond" but it's a lot more like throwing electricity through a dead body than working with a competent duet partner.

FWIW, it's not really meant to be playing alongside you. The idea is for you to play a short tune, then wait for the AI to respond with a tune of its own based off of yours.

Well, FWIW as well I tried exactly that as well and it was like it could barely follow what I did (rhythm was off) and then it went in other directions that were not pleasing to the ear. Nor made much of any sense musically. So, again, it's like a cat or child was mimicking my motions just enough to make me dislike the result.

Haha...love the cat and dead body idioms. Captures the fail perfectly.

This is fun. It actually feels like I'm taking turns with someone else on the other end of the screen who takes what I play and sends back a riff on the notes. The trick is (as the little intro shows) to only play for a few seconds, don't play simultaneous notes, and then stop and let the AI respond. Once the AI has finished, you can try again. If the AI starts playing before you're done, you've screwed up and should make your little sequence of notes shorter. Basically, the title of this is misleading... if you keep playing expecting the AI to join you in a simultaneous duet, it'll sound like cats fighting inside a piano.

Try playing the "Dukes of Hazard" horn: on your keyboard, press (without overlapping the keypresses) "LJGGGHJKLLLJ" and let the AI respond. More often than not, it'll come back with a vaguely pleasing riff on that sequence using mostly the same notes and in the same tempo.

My immediate reaction was to try and confuse it, so I played "Roll Call" from the Thomas show (kids are obsessed !) It's a bit chromatic so I thought it would have trouble, which it did.

Next, I tried putzing around on the white keys but made sure to never hit a F or F#. I wanted to see if it would try and guess which key I was playing in, C or G major. It seemed to guess C most of the time, though one time I tried hinting at G by starting out on G, it got confused and starting playing both F and F#.

I was just clicking with a mouse, so rhythm was hard to judge but overall this thing did a really bad job "complimenting" my playing!

You can use your keyboard, the middle row maps to the white keys and the upper row maps to black keys. Works better than a mouse for tempo, IMO.

Are these standard keys for some other piece of music software?

I keep getting confused because I'm so used to the old-style tracker keys (I keep hitting J for Bb and getting B, which isn't helping my C minor improv). Also one octave isn't great, lag is killing the rhythm, and it keeps not waiting for me to finish.

Neat idea though.

(I'm curious to know whether the training set for the version shown in the demo is the same as for the live web version, as the demo sounded way better than anything I'm getting out of it - though obviously they would have cherrypicked the results for the video.)

It seems to be the same keyboard mapping that Reason uses: https://youtu.be/7maQ1zjvb-M?t=28

If you are interested in real-time generative music, check out IRCAM's project SoMax/OMax: http://repmus.ircam.fr/omax/home

Sample (software is piano trained with Bill Evans corpus): http://www.dailymotion.com/video/x2j39hf_somax-with-remi-fox...

I played the A-section for Heart and Soul (the song from the movie "Big") and its response was enough to tell me it was not going to be playing actual duets.

I'd be interested in a version that has learned accompanying portions to many popular melodies. If I could play the right hand part of a piece and it played the left hand I'd be very impressed -- also this would actually be useful software for aspiring pianists.

I think Google can learn something from Microsoft here: [1]

[1] https://www.youtube.com/watch?v=3oGFogwcx-E

just tried this on my midi keyboard( web midi is awesome!) but the "AI music" response was very underwhelming and sounded bad.

I would be interested if "AI" can help me with different timbres based on the physical environment that music is playing using various sensor like air pressure, humidity, room resonance, maybe even heartbeat, perspiration ect. Make that a new dimension of information coded into the song along with pitch, timing, amp and timbre.

I'm not sure if it is a reflection of my own lack of understanding of music, but it sounded like chaos to me. Garbage in garbage out though.

I was excited to play with this, but ultimately left disappointed. I was hoping that my initial melodies would be expanded upon and that I could build a series of riffs that would become increasingly complex but based on the same starting melody, but in the end, the responses from the AI seemed to be fairly random, and didn't seem to be built upon the previous sequences at all. The AI also changed keys a couple of times, which was disconcerting.

To whoever implemented this AI: Can you actually play music yourself? Your AI can't. I tried to play “Satori Maiden” (link: https://www.youtube.com/watch?v=0c9bea_thaM ), and your AI only replied with unmusical nonsense.

Go easy man. It's an "experiment" for a reason.

I'm opposed to this trend of trying to replace actual understanding of the problem at hand with massive amounts of computation. If you want to write a program that writes music, then learn how music is actually written first.

---

@hellofunk: If you want the ML algorithm to actually learn how to write music, at the very least, the programmer must be able to express the goal of producing musical sequences of notes.

It's opensource bro. Translate your problem to a requirement and impress us.

Look, it couldn't even follow a basic tempo when I was playing. That's not even close to being a "duet" as described. Maybe whoever wrote it has a limited grasp of English, because that experiment is a far, far, far, far, far, far, far cry from a "Duet" experience.

Well, one where you don't look over at your partner and wonder if the have any clue how music works before pushing them off the piano bench.

That's too many fars.

Why would I? While I use computers to play music all the time, I don't think using computers to write music is a good idea in the first place.

---

@jgalloway___: I'm not sure if I'd call it an “expectation”. I just dislike it when people throw random shit at the wall and see what sticks. When you write music, you should first think how it's going to sound, convince yourself that it's plausibly good-sounding, and only then actually play it just to make sure it sounds good. If you do it the other way around (first play random stuff, then filter out what doesn't sounds good), you're a wanker.

---

@ohitsdom: I'm totally fine with improvisation driven by ideas, not random computation. Coming up with ideas on the fly is good. Passing noise as genuine ideas is not good.

Ahh.. you have an expectation. Is that conducive to an experimental process?

That's fine that it's not your style, but experimentation is a huge source of new music and reinvented ideas. Dismissing any music that wasn't strictly planned seems a little silly.

You can grade experimentation exactly on its ability to source new music and ideas.

This experiment does neither particularly well.

> then learn how music is actually written first

Is not the idea to see if the ML algorithm to learn how music is written?

I thought the idea was to apply ML in the least intelligent possible way and then say "Well, at least we used ML, and that's cool, right?" when the results are garbage.

I agree with catnaroek - this is a bad example of human learning, never mind a good example of ML.

The problem is that this experiment might as well just replay a random collection of the N most recent notes played, with durations shuffled around, for all the musicality it demonstrates. He wired a neural network to some midi samples, and it didn't generate anything of use. Yeah, it was an experiment, and there's value in failed experiments, but you don't typically make a big website to present your failed experiment.

Maybe if you could set it to "Blues" or something like that, switching training sets, it would do better, but as is it's a nonsense generator.

There's nothing wrong with experiments that eventually fail, but there's something wrong with carrying out an experiment that you can tell in advance can't possibly succeed.

For those looking for harmony try letting the machine lead .. if you mimic it's style and complement with a hint of your own rhythm it can be satisfying.

This is cool! I've been trying to use ML to automate the entire production pipeline. Choose chords, synths, loops, mix and add FX. http://signalsandsorcery.com/

Im finding extracting audio features from loops the hardest part. The selection of notes is actually not as difficult as choosing a good timbre for the notes.

Style transfer and generative adversarial networks have worked well with images. How can these be applied to music?

It does seem like an interesting edge to explore doesn't it? ML techniques combined with spectral analysis could yield some interesting outcomes. The datasets wouldn't be too hard to find either - most people have pretty extensive digital music libraries.

Is that how they did it? Or is it just training some RNNs to repeat the same style back?

Sony has been working on style (genre) transfer in music lately: https://youtu.be/I9M8l2guPSo?t=21m30s

I know this is meant to be a "cool experiment", but from trying this out for a while my impression is:

I could just build a random sequence generator and tell the world "I made this with tensorflow" and people would believe. I mean, none of the response sounded like music. Which is fine, but then again, what is the machine learning for, if it just sounds like random keystrokes?

I think that this response is confused. What is your point? That people can't differentiate random notes from a duet? That people would believe that anything associated with tensorflow was imbued with sophisticated behavior?

How does that relate to this experiment? It's clearly not just playing random notes in the example video...

Make me think about the movie "Electric Dreams"

https://youtu.be/aVE8taDWmMc

I want to hear one trained on Monk, Bill Evans, etc.

I get the feeling this one was trained on Cecil Taylor.

Even more sophisticated duets with a computationally creative computer: CIM Sessions 2017. https://www.youtube.com/user/QCGInteractiveMusic/videos?sort...

Even more sophisticated duets with computationally creative computer: CIM sessions 2017 https://www.youtube.com/user/QCGInteractiveMusic/videos?sort...

I've had the most success in my own experiments using many dedicated Models. The first and most important model is a NN that generates a chord progression. After that is much easier to select good notes! http://signalsandsorcery.com/

Markov chains might do better for progressions :)

I have tried just using N-grams which is pretty good but it doesn't have any structure. My next experiment will be to use the 'word2vec' algorithm the same way it it used for NLP. Meaning first create relationships between chords then create sequences with RNN.

As an experiment I think this is well done. It's fun to mash the keys and see what happens. If there was a feedback loop to register if the user was happy or displeased with the results via facial recognition or other input with expanded training data it could become quite interesting I think.

Thanks for this!

I'm getting some good results but I'm also a clasically trained musician so if you haven't studied music it may not give you a pleasing result but if you are a trained musician you may get more mileage.

I think it'd sound more interesting if the computer compliments what I play. For example, if I play in C key, the computer tries to guess the chord progression and play the rhythm that matches what I'm playing.

I believe ML can do better than this.

I tried this at Google Tensorflow Summit and it was fun!

Reply on news.ycombinator.com