What makes Bach sound like Bach? New dataset teaches algorithms classical music
washington.edu
washington.edu
The actual thing they're reporting is:
'“You need to be able to say from 3 seconds and 50 milliseconds to 78 milliseconds, this instrument is playing an A. But that’s impractical or impossible for even an expert musician to track with that degree of accuracy.”
The UW research team overcame that challenge by applying a technique called dynamic time warping — which aligns similar content happening at different speeds — to classical music performances. This allowed them to synch a real performance, such as Beethoven’s ‘Serioso’ string quartet, to a synthesized version of the same piece that already contained the desired musical notations and scoring in digital form.
Time warping and mapping that digital scoring back onto the original performance yields the precise timing and details of individual notes that make it easier for machine learning algorithms to learn from musical data.'
It also mentions that they attempted to apply existing deep learning algorithms designed for speech recognition to their new dataset, hoping to be able to accomplish a task such as predict a single missing note from a long string of notes. It does not say whether this worked.
When we talk about "what makes Bach sound like Bach," the technical concept we have in mind is the recent work in computer vision on style transfer. For example,
https://arxiv.org/abs/1508.06576
We are excited to work on adapting these models to the musical domain!
As for note prediction, you can see our results in our paper:
https://arxiv.org/abs/1611.09827
Our are results for simple (2-layer, not very "deep" models); we were interested in understanding the low-level "features" of music rather than building a model that maximizes performance. Nevertheless, the results are quite promising; I'm confident that someone using our dataset with a deep network and a lot of gpus could blow our numbers out of the water! :)
Tutorials on how to set up and evaluate this task are available on our website:
I am starting to record my own dataset for solo jazz piano - all midi though. Monophonic melodies, and matching chord voicings and voice leading from one chord to the next. With the goal of learning to generate a good sounding jazz piano arrangement to a given melody with nothing except monophonic input.
Style transfer is good at essentially texture transfer - I suspect it won't work that well for understanding music theory (or text), especially with long time series dependencies, but will be very curious to see what emerges.
I'd like to hear more generative music samples from DeepMind's WaveNet too, the piano samples they published sounded very good, but it was unclear what the model had learned or generalised - and how much was semi-randomised recall. I haven't seen the open source implementations of WaveNet produce as good results yet - probably because it's computationally very expensive to train and run, and that limits experimentation. I saw Aäron give a talk on it a couple of weeks ago which helped me understand the stacked dilated convolutions - but would still like to hear more music examples :)
The style transfer stuff comes later and as you observe, we'll probably need some new ideas to make that work well. I haven't thought about this deeply yet, but my intuition is that maybe instrumental timbre is an audio analog of visual texture, so maybe a reasonably direct "port" of style-transfer to the audio domain would let us construct demos that, for example, rewrite a cello recording to sound like trombone.
Let us know when your dataset is complete! I love jazz.
However a counterexample from my own life: I only learned German starting in my late 20s and when, 10 years later I heard the Matthäus-Passion and could understand the lyrics I wept...and I don't even know much about christianity.
Which parts made you weep?
Start with "Eli, Eli, lama sabachtani" and "Wahrlich, dieser ist Gottes Sohn gewesen" and "Und von der sechsten Stunde an ward eine Finsternis über das ganze Land bis zu der neunten Stunde" and...see? I can't stop once I get started.
It even begins with him being a bit of a prig as his acolytes express their unhappiness. But then he himself expresses pathos (Meine Seele ist betr ̧bt bis an den Tod, bleibet hie und wachet mit mir.) The story becomes unfair. I really felt the unfairness of his trial.
It's a much more sympathetic story than, say, Plato's recitation of the death of Socrates, who comes off as a jerk.
It's actually not true that the "vast majority" of Bach's work had no texts. Bach wrote over 200 cantatas (with around 5-10 movements and separate texts each) plus an assortment of masses and 4 extremely large choral/orchestral works: the St. Matthew and St. John Passions, the Christmas Oratorio, and the B Minor Mass.
Looking at the catalog of all of Bach's works, which I have here, BWV (Bach Werke-Verzeichnis) numbers 1 thru 1071, you get all the way to BWV 525 before you even get out of the vocal works. Numbers 1 thru 524 are all cantatas, masses, oratorios, lieder, the many chorales of course, secular and comic cantatas, etc. And of course many/most of these are far larger than the individual organ works.
(Bach actually wrote a lot more cantatas than this; but only around 224 of them survived. Another hundred or so were lost.)
So to me this seems more directly applicable to transcription (ie. taking audio and turning it into sheet music) or synthesis (taking sheet music and turning it into audio of a human-sounding performance) than it does to composition or finishing unfinished works by famous composers. The output of the compositional process is generally sheet music, not audio, so it seems to make more sense that problems around composition would be trained and learn in the sheet music domain.
I'm not a machine learning researcher though! This is just my impression as a musician.
I'm one of the authors on this paper. You're right that the most direct applications of this dataset are transcription and synthesis. One of the cool aspects of end-to-end learning models is that they discover a "representation" of data that can be useful when applied to other tasks. We speculate about some tasks like recommendation and composition on our website:
http://homes.cs.washington.edu/~thickstn/musicnet.html
We're also interested in music like jazz and pop, for which good scores are often unavailable. Classical music is nice for training models because we can use sheet music as labels to learn a representation. Many aspects of this representation, such as rhythm and harmony, may transfer to other musical genres. Learning about classical recordings could bootstrap learning for other kinds of musical audio.
So while you're right that it's probably easier to learn a model to complete Bach using symbolic sheet music, we feel that addressing complex tasks directly from raw audio is worthwhile!
http://www.markheadrick.com/midi/absmfaq.txt
In section 1.4 they very emphatically state that "with current technology, IT CAN'T BE DONE."
They conclude: "Think of it this way: If you don't mind spending more than the US national debt on computer equipment and waiting a few years for the job to complete, you can have a system that MIGHT accurately convert the digital waveform data of a 5 minute song into a small, compact MIDI file.
Otherwise, you can blow a couple of thousand dollars hiring a professional band of studio musicians and engineers who can probably give you what you want in about one day."
It is humorous for its emphatic-ness, but also educational for being a picture into how we've historically thought about this problem.
As a composer, the coolest potential I see here is training a model to create realistic mockups from MIDI compositions. For that purpose, though, it would be better to start with a fully monophonic/solo-instrument dataset, which would simplify the learning. Also, MIDI data is not entirely sufficient: annotations on dynamics and playing technique would be necessary to make a good mockup tool, since this is the kind of information one might even give to human performers.
Anyways, it would be tough for such a tool to catch up with current state-of-the-art, sample-based mockup tools, which are already baffling in their realism, although they usually require a lot of work to get good results. But one can always dream of a "Stokowski" or "Karajan" neural network that interprets your MIDI composition with emotion and sensibility!
Another problem is once you have the music there's a tremendous amount of "interpretation" that a musician does. the nodes may each read 1/8 but a musician might add or subtract 1/64 has he/she feels is good.
other times the change is more mathematical 1/8+1/8+1/8 might have to actually be read 1/12+1/12+1/12 = 1/4 but that is much easier to fit into a computer
I have said nothing of dynamics (loud/quiet), articulation (stoccatto, slurring etc).
scores are available in IMSLP and other sources. but are computer files available as well?
Here are some collections of music in that format:
http://www.global-supercomputing.com/people/kemal.ebcioglu/p...
Unfortunately I can't seem to find the samples now, but to my (untrained) ear they sounded as Bach as the real thing.