Omnizart: Library for automatic music transcription

Wow the Onsets and Frames algorithm is insanely interesting. It's like a mixture of run-length encoding of (vertical and horizontal) strings of (0dim/time) and (1d/time) structures (onsets as points in time, activations as lines in time). But..hm.. why stop at such low dimensionality structures..! :^)

I just tried this and it works very very nicely! Thank you for sharing!

I just tried this with Sexy Sadie and the result was awful.

There has since been follow-up work to extend Onsets & Frames to multi-instrument music: https://magenta.tensorflow.org/transcription-with-transforme...

How do you get a midi?

Listening to the input and output, the reproduction is like comparing a cat to a picture that resembles a cat, but isn't a cat.

Awesome, thanks for running the test!

I can't help but feel it is heavily impacted by ambience of the recording as well. The midi is of course a very rigid and literal interpretation of what the model is hearing as pitches over time, but of course it lacks the subtlety of realizing a pitch is sustaining because of an ambient effect, or that the attach is is actually a little bit before the beginning of the pitch, etc.

If it could be enhanced to consider such things, I bet you would get much cleaner, more machine-like midis, which are generally preferable.

That is a tremendously big “if” considering how many times that problem has been attempted. Even just detecting the key of a song is awfully fuzzy.

the result can be used as audio fingerprints, which is not a new thing. This has something to do with how things like Shazam work.

In the field of speech, you have the concept of forced alignment, aligning an audio recording with a transcript by way of text-to-speech and pattern matching, which can certainly be used to detect some sorts of errors consistently and various other errors with some probability. (I’m not up to the point of doing this, but I’m early in making software for audiobook recording and plan to see what I can manage in error detection eventually. Unfortunately from what I’ve seen just with a little bit of casual playing around, small errors are not generally well identified because there will be too many false positives.) You could probably adapt a lot of that stuff to music without too much trouble, though it’d certainly require changes and integration effort. My vague feeling is that this direction, with MIDI rendering and pattern matching, would probably be surprisingly effective and I suspect better than trying to more directly match the recording to the score with the help of just Fourier transformations, or converting the audio to a score with something like this and comparing them, though even with the forced alignment adaptations I imagine you’d be changing it to use mostly Fourier transformations for feature detection.

(I’m a good programmer in general, but still decidedly a layman in audio matters. Take my thoughts here with a grain of salt.)

This is what Yousician does for singing (and other analog instruments) but unfortunately not for the violin: https://yousician.com/singing

Haven't tried it but maybe the following app could help?

> Trala uses signal processing, groundbreaking technology that analyzes the sound of your violin and gives you instant feedback on pitch and rhythm every time you practice. When you play the wrong notes, we’ll help you get back on track.

https://www.trala.com/ (I just remembered it being recommended somewhere.)

I think what you're describing is the SingStar game on PS3 :D We used to play it a lot with friends. Truly fun.

Check out the Rock Band video game series, they had a really nice (non automated) version of this: https://youtu.be/pV9_O9bKiLg

There’s plenty of video games that do this going back to 2003’s Karaoke Revolution - if not earlier. Plenty of other games and series followed suit including as Rock Band, Guitar Hero, Lips, SingStar. There’s plenty of mobile apps that work similarly. AFAIK they basically track your pitch and timing. I remember Lips in particular rewarding vibrato as well.

The algorithms as I understand them are grading you based on sustained pitch (adjusting for octave) and cadence. Generally they are easy to fool - you can basically say gibberish but as long as it’s within the expected parameters you’ll still get a good score.

They are still rather fun though if you enjoy that sort of thing.

Well, usually (?) it means turning a piece of music (audio) into written sheet music/a score. It means that in jazz and...I believe many kinds of contemporary music. (In classical music I've heard it applied to e.g. turning an orchestral score into a piano score, i.e. an arrangement.) Here they appear to just make MIDI files, avoiding the problem of how to write the notes most readably, which is a significant part of transcription, in both these senses.

All the examples given here, though, appear to be of a super-simple variety, with dead-simple chords, all notes with robotic, mathematically-simple timing over fixed tempos produced apparently by drum machines - like toy music, the kind of music that's no challenge at all to transcribe, and in the real world I wouldn't bother transcribing by hand as there's nothing to be learnt by doing so, as you can hear exactly what's going on without it. So, that's weird.

In this context it means listening to music and writing it down in music notation.

To transfer / scribe something that you have heard. Just like a “transcript” is the written record of spoken words, transcription is the act of transferring something heads into written form.

Transcription is a valued skill among musicians. The ability to hear a piece of music and write it down can be more than a little more difficult than transcribing speech.

Not sure if apocryphal, but Mozart is known an expert transcriber, able to hear an orchestral work and write down all the different instruments as they played their parts on just one listen.

Scribe means to write, and trans- means across or over

In this case they mean to take music (audio) and write down in musical notation everything that’s being played.