Comment by crazygringo | Hacker News Reader

Comment by crazygringo | Hacker News Reader

Parent Full thread

This is really clever!

I love whenever it's possible to upsample/restore media due to known constraints in the original -- in this case, how screens work.

Something analagous I've been waiting for is regenerating old scratchy piano recordings. Piano is unexpectedly simple compared to other instruments -- the only inputs are really note down + speed, note up, sustain pedal pressure, and (less frequently) soft pedal pressure.

Seems like you should be able to turn any solo piano recording, no matter how degraded, into a relatively lossless MIDI representation, then re-record that replayed physically (via motors, which exist already) on a modern piano, or even just synthesized, trying to be as true to the original piano's characteristics as possible. Losing literally none of the artistry.

It seems like this should be "easy" for piano in a way that it isn't, for example, with violin which has so many more complicated characteristics of pitch, timbre, bowing, vibrato, etc.

This has been done in various forms! One interesting one is shown in https://www.youtube.com/watch?v=6hv2zh_Z0Io Sergei Rachmaninoff recorded a 78 album simultaneously with a piano roll of several pieces - thus capturing the 'keystrokes' and the audio of intonation of the master's hand. The piano roll was converted to an automatic reproducing piano (super high end player piano, a Bosendorfer 290SE) and massaged by an expert to sound almost exactly like the 78. Then it was re-recorded in modern fidelity, playing back from the 290SE. It is exciting to see and hear a 290SE (re)play in person, but a little weird in a concert setting with no pianist to watch.

That's amazing, thanks! I was even thinking that Rachmaninoff's recordings would be where I'd start.

It's like a ghost playing. Absolutely crazy to hear his touch but with modern fidelity.

When you translate to MIDI, you are going to lose a lot of the subtle pitch and tonality of the piano. One string vibrating produces overtones and a chord will vibrate in a very deep way on a grand piano that I feel you cannot replicate in a MIDI, and therefore that detail would be subsequently lost.

For instance, how long you are touching the string - while you're touching the string, there is a sound - but after you let off, you get the "reverb" - and there is different reverb for how you hit the key, if you bounce, or if you stay for a split second longer for staccato, I don't feel like these subtleties translate to MIDI.

It is certainly easier than violin, that I will grant.

edit: IMO the best way to do what you are describing is get a really good pianist to sit down and do the work. I don't think that (current?) machine learning can really "understand" the nuance of phrasing esp that would be coming from older recordings.

> I don't feel like these subtleties translate to MIDI

But wouldn't they be reproduced when replaying the MIDI data physically on a piano?

Ultimately isn't how you hit the key and bounce/stay still just initial velocity and then timing of letting go? Perhaps the velocity of letting go would have to be added as well, but I'm not actually sure if that's really acoustically meaningful.

I guess I don't see why all the reverb and ultimate sound complexity wouldn't be recreated in playback? Of course, this requires actual physical playback on a similar enough model of piano, or else a synthesizer that is sufficiently accurate.

Well - for one thing there is the pedal is not itself binary but in degrees - and there are three pedals, one of which if you depress, will silence only some of the strings. I don't think MIDI itself is capable I guess, some other format might be. There are a lot of factors, and pianos sound different from each other, I think that would be lost.

MIDI is definitely capable -- code 64 is used for sustain pedal, and code 67 for soft pedal. And it's associated with a byte value for how far the pedal is depressed.

The third (middle) pedal in pianos is nonstandard -- i.e. used for different effects on different pianos, whether sostenuto or bass damper or practice mute.

In actual performance the only time it's ever really used (and rarely at that) is as sostenuto, since that's what it does on grand pianos like Steinways, but its effect is indistinguishable from simply holding notes for longer durations, so MIDI can simply represent its effect that way. (Unlike the sustain pedal which increases resonances in a big way and needs to be represented independently, or soft pedal which changes timbre as well as volume.)

A piano can be intentionally and or unintentionally not in tune. Every Piano has its own unique sound (due to manufacturer and also form factor) and is played in a place where temperature can have another affect in the sound.

But it looks like already people have attempted the transcription strategy you describe https://www.google.com/search?hl=en&q=machine%20learning%20n...

Reply on news.ycombinator.com