Hacker News Reader

That's an interesting comment. So, the midi->score conversion is indeed one of the harder parts, I spent a lot of time on that and I'm not yet satisfied with the results. There are quite a few ways in which this can be improved further, there is a long list of 'todo's' where I basically have worked out how to make the improvement but I have not yet gotten around to actually implementing it.

Midi is 'lossy' in the sense that it flattens a bunch of knowledge about how a piece is to be rendered into note-on/note-off pairs. You then have to do all kinds of tricks to even begin to approach what the score looked originally. Lots of enharmonically (identically sounding) combinations may have looked completely different on the original score, and when you start adding the various chord combinations it gets even more complicated. Even so, I'm strongly convinced it can be solved and we have made considerable progress on this already.

The 'wiggle room' aspect of it has been addressed to some extent. For instance, we build up a map of all the notes that are sounding at a given time, and will ignore notes that are technically 'wrong' but that are still sounding. This allows for much smoother restarts after mistakes.

Your final point is a bit tricky, but thank you for the hint, I will definitely add it to the list of things to look at to see if we can find a way to make that work.

Musescore does do a great job and is open source, so there's that. Another option is to have users manually upload midi with a proper score attached to some store or community repo with user reviews. It's less satisfying as a solution, but it means all it takes is one motivated user for everyone to enjoy the score.

For my final point, you can model this as a hidden markov process. You're observing notes, but not the user moving his mental "cursor". Classical filtering algorithms should do well here.

Ah, one more missing feature, good scores have fingering, but it gets lost in the midi. I'm pretty sure fingering can be infered though and this would be pretty powerful. Amusingly, it can also be seen as a filtering problem:

- At any given point, a hand can have a finite number of fingerings, and that number isn't that large. Give each of them a difficulty.

- Now, make that difficulty conditional on the previous hand position.

- Run Viterbi's algorithm on the score to minimize the difficulty.

The difficulty can be obtained from an expert or through some physiological model, but they could also be inferred from an existing body of music sheets.

Fingering we have a pretty good hold on already, so I'm not too worried there.

Importing alternative formats (xml for instance) is definitely an option worth looking into, but because the bulk of the material that is out there that people can freely access is in midi format we chose that one first.

> have users manually upload midi with a proper score attached to some store or community repo with user reviews. It's less satisfying as a solution, but it means all it takes is one motivated user for everyone to enjoy the score.

Musescore already has such a collection of sheet music https://musescore.com/hub/piano so it'd also be possible to allow users to import from there, going from sheet music to midi instead of trying to reverse the process.

Yes, but then you would still lose quite a bit of information. Besides that, musescore is pulling a bait-and-switch on the user generated content that makes gracenote look good.

> Midi is 'lossy' in the sense that it flattens a bunch of knowledge about how a piece is to be rendered into note-on/note-off pairs.

As using MIDI will probably never be perfect, have you thought about using the score in a suitable format, like LilyPond, directly?

Yes, absolutely, and we will add more import formats.

I don't really follow the idea that "midi is lossy" in the sense described. Because it uses note-on/note-off pairs? I don't see the problem there. Certainly the idea that a score somehow has more information in it than a midi recording is ridiculous. A score always requires interpretation. The midi is, in contrast, fully specified, very little room for interpretation.

A midi recording contains pretty precise velocity info for each note (0-127) that is either not present at all in a score, or is present only in the form of a limited number of dynamic markings[1], which govern the flow of dynamics imprecisely, and almost not at all on a per-note basis.

With tempo it is similar to dynamics: midi will precisely determine when notes turn on and off, down to the millisecond, in the process resulting in the midi performance having a precise tempo. In contrast, musical scores have limited means to indicate proper tempo[2], necessitating (as with dynamic markings in a musical score) much interpretation by the musician.

All of this is not to say that a midi performance is somehow "better" than a score. There are good midi recordings and bad ones. A good recording will generally be made by an accomplished musician who in making the recording records fairly precisely a particular (and good) interpretation of the score. Likewise, a bad recording would generally be the result of a poor interpretation of a score. But whatever the case, the midi recording is far more precise than the score itself.

[1] https://www.aboutmusictheory.com/music-dynamics.html

[2] https://en.wikipedia.org/wiki/Tempo#Basic_tempo_markings

> Certainly the idea that a score somehow has more information in it than a midi recording is ridiculous

A midi file is designed to contain different kinds of information than a score, so it can absolutely contain less information (of a certain kind) than a score.

Imagine a group of notes that are attacked very close together in time in a midi file - is it meant to be a chord played without arpeggiation? A chord arpeggiated as fast as possible? A chord arpeggiated to a specific tempo?

When a composer writes (say) a chord with an arpeggiation sign next to it, it specifies that the notes are to be played in a certain order, and quickly. There are multiple possible performances that comply with this "specification". MIDI cannot encode this information (the information being the set of acceptable performances).

Thanks, yes, I would agree that a midi recording and a score to a large extent encode different types of information. The score has less of some of the kinds of information that are in fact present in midi. And the midi, of course, doesn't specify anything about ranges of interpretation; it simply represents a specific interpretation of a score.

Midi note code 61: is that a C# or a Db, you pick...

Piano Marvel allows you to add your own pieces to the library, I did it with one I found online. What they recommend is using Musescore to convert the MIDI to a score, fixing it up in Musescore and exporting as MusicXML. Musescore did a creditable but not perfect job of the conversion, but fixing it up was easy. Then you import both files into Piano Marvel. I'm actually not sure why they need the MIDI at all once you have the MusicXML, given that Musescore can play the music from the score directly.

I think what's missing on most "piano-games" is that it doesn't really mirror the way I learn a piece.

The way I learn is by bars, repeatedly re-playing bars say, 1 to 4, until I get them, them 3 to 7, until I get them, etc.. But most piano apps want you to start playing from start to end slow, then faster, etc.

So: piano apps should allow breaking a piece into bars, allow you to repeat those bars again and again until you exceed certain score. Maybe even keep track of which bars are harder to play, and repeating the difficult bars more often, etc.

I’ve been using piano marvel which does exactly that, and find it pretty great. The only issues with it are that it’s built on somewhat old tech and can be a bit buggy, but for actually learning pieces it is genuinely useful (I’m around grade 7) and also has method/technique pathways you can go through. I think there’s a free trial for it too.

(Not affiliated, just something I personally use)

This one actually does exactly that, so please give it a try. It also has an auto mode where it measures how well you play and then sends you to practice those bars where you are still weak.

sg! will take a look!

> you're playing at a given pace, miss, stop and repeat a bar, the app should detect that automatically instead of just carrying on expecting you to keep playing.

I'm a piano noob but wouldn't you want to practice catching back up with the beat if you ever intend to perform?

This is a really good question, so I don't know why there are downvotes. There are 2 scenarios: 1) you're trying to learn the notes and 2) you're trying to get it in shape for a performance. It sounds like this software is geared towards scenario 1 where you don't yet know the piece and want to learn it. Generally when learning the piece it's best to fix mistakes as they happen, otherwise your brain sort of subconsciously assumes that was what you intended. Once you have the piece down, then when you make a mistake you want to keep going and pretend it never happened because otherwise the rest of the band will leave you behind. (And for soloists, nobody wants to hear you go back and correct a mistake.)

Yes, you really don't want to end up 'practicing your mistakes'.

Notes should directly align with the keys for usability. Example: https://www.youtube.com/watch?v=6ZJTdQ-xCBE

I'm not quite happy with Synthesia. There is no visual feedback on completing a note. This has been a staple feature in Doremi Mania and Keyboard Mania for twenty years, so they have zero excuse for not implementing it.

Aside from the relatively difficult problem of creating scores from MIDI; when practicing piano as a beginner it'd be interesting to have help in positioning/finger selection. I wonder how hard it would be to compute fingering plans (maybe even with a wingspan parameter).

We are working very hard on solving this in a very interesting and beautiful way. Stay tuned.

I’m super curious, if you don’t mind my prodding. I looked into doing this for my own version of a JavaScript midi teacher a while back. Any chance you’re doing it via the neural nets sequence to sequence translation algorithms that are super popular in NLP right now? I think that’s a practical way to do it, except for needing a training corpus. I was going to reverse engineer synthesia’s data to try it out, for what’s worth...

We do not think that it will need a neural net in order to be able to do this, but we need to prove that in order to make it stick and we're not quite ready to do that. Keep in mind that this is in every sense of the word a side project, we only work on this when we do not have technical due diligence work for customers and that still continues quite strongly. But on 'off' weeks we work on this and the project progresses quite rapidly when we do.

I think it could be done with a constraint solver, with a lot fewer cycles.

This is the route we are looking at, with some twists.