Piano on Paper | Hacker News Reader

Piano on Paper

github.com

225 pointsthunderbong63 comments

I’ve often wondered about the reverse. Get the finest grand piano action, like the ones used in the Fazioli, and create a minimal mechanical unit that has just that piece and the pedals (with the action mechanics that connect the left/right pedals to the keys) and then nothing else but a couple of sensors and a couple fast USB-C. The Kawai Novus series did an OK action and left pedal, but then they connect it to a whole furniture and their sound synthesis and vibrations are imperfect. With modern powerful GPU the sound should be possible to become near perfect and easy to tune to one’s preference.

I am no master pianist, but I imagine much of the feel of such an instrument comes from the physical weight of the keys, and the feel as they strike the strings. Although good quality keyboards have "weighted" keys, they don't feel exactly the same. I think it would be hard to reproduce this exactly.

I agree. Perhaps I was unclear. I was hoping that someone can use the exact action of the grand piano, but skip the strings, soundboard and so on.

craigslist is full of free upright pianos and occasionally free baby grands. Of course if you show up with a chainsaw instead of 5 men and a dolly...

Silent grand pianos exist, though the ones I'm aware of also include the acoustic bits as an option, e.g.,

https://usa.yamaha.com/products/musical_instruments/pianos/s...

https://www.boesendorfer.com/en/pianos/silent-edition

The top-end Yamaha electric pianos have a near-complete replica of the action inside of them. If you adjust the volume exactly right, you almost replicate the feel of playing a real piano.

GPUs may not be great for latency.

You may have a point. Perhaps naively I imagined that if competitive gamers can handle it, they may be useful enough for piano. Currently the CPU software is lacking in quality (though improving over time) but has acceptable latency compared to grands (which also have latency due to the mechanism and the location of the sound source).

Real (proper) pianos also have latency between a finger pushing on a key and sound coming out. There's mechanical bits that need to move some distance before a string is struck or muted, and these bits do not have infinite acceleration -- they've got mass, momentum, and inertia.

Can these latencies not be balanced between techniques?

I'm no expert on synthesis or pianos, but my immediate thought is that the digital thing may need slowed down to match the latency of the analog thing.

Kinda like a silent piano? There are aftermarket options like PianoDisc

https://en.wikipedia.org/wiki/Silent_piano

Yes but without the need for the full furniture. One should be able to miniaturize everything else but the action. I think such compact designs will eventually appear but the difference in real action versus very good hybrid digital actions is of highest interest to musicians who typically have access to real pianos. Roland, Yamaha, Kawai, and others have been moving in that direction over the years and the top models by Yamaha and Kawai have passable actions though they still have a large piece of furniture around them.

Very high end digital pianos have an action that is quite similar to a real grand.

This is a relatively cheap digital piano action https://www.youtube.com/watch?v=S_VT0IpM66k

These are 2 versions of a mid-high range digital piano action https://www.youtube.com/watch?v=rJEAKI2jBLU

This is a very high end digital piano action https://kawaius.com/wp-content/uploads/2021/06/Kawai-NV10S-M...

That very same action is also used in acoustic pianos, and the acoustic version looks like this (you can see the main difference is that the hammers have felt) https://snadenspianos.com.au/wp-content/uploads/2018/05/MG_9...

And this is another (more traditional) acoustic grand piano action https://www.youtube.com/watch?v=v5O1Kfd_4Fw

The kawai vpc is pretty much that, right? You don’t need fast usbc for audio and it’s the best action on a midi controller

A large part of the experience comes from the actual sound. GPUs don't matter. Mozart has some very good physically modeled pianos that respond very naturally. But to get that to sound like a real piano in a real space is a greater challenge than reproducing the feel of Fazioli.

Obviously a bit rough, but a start. This is the kind of content I like on HN.

yeah! it was a hella challenging to detect touch.

I had an idea for a similar project. Basically, a tabletop drum kit. In this case, you’d tap on a few objects on the table that make distinct sounds and then map that back to a virtual drum kit in your DAW using midi. I never got around to building it, but I am still keen on the idea.

That already exists in some forms, most notably perhaps in Dubler 2. I’m also working on an open source version of something similar, albeit quite involved and aiming at professional real-time performance and drumkit augmentation (if all goes well to share on HN this month for the first time :))

Hey that is cool! lemme know if you need any help, send mail if required!

Congratulations on shipping!

You might find this interesting, a comment on https://news.ycombinator.com/item?id=35376138 (https://news.ycombinator.com/item?id=35379983):

> I had been playing with the idea of creating a browser-based virtual piano for when I'm travelling and don't have access to a real piano but have my laptop with me. The idea would be to point the webcam down at the table between me and the laptop, and play on the table as if a piano were there. Then use the mediapipe framework [1] to capture finger positions, and use those to update a virtual environment like the one you have here.

> I put it on hold due to the significant engineering required, but it seems you have already implemented (and open sourced!) the browser-based piano simulation component.

> A quick scan through your repo indicates that this is all implemented in Python. I see that you are using mujoco_wasm [2].

[1] https://google.github.io/mediapipe/

[2] https://github.com/zalo/mujoco_wasm

Reminds me of virtual laser keyboards, eg: https://nerdknowbetter.com/best-virtual-laser-keyboards

I'm surprised they actually exist instead of being a rendered design concept that I saw many years ago, and are apparently common and numerous enough that someone would write a listicle about them. They don't seem to cost more than a good regular keyboard either, but I've found reviews saying that the typing experience is about as uncomfortable as you'd expect from tapping on a hard surface.

The typing experience is <chyrosran22>Hhhhhhhideous</chyrosran22>[1]

[1] https://www.youtube.com/watch?v=1Cy1dPlo8No (NSFW)

You don't need to type on a hard surface (but anyway, most people tap away on their phone screens for hours in a day). You can type on an octopus if you want. Anything can be a projection surface keyboard with enough inspiration.

This is a delightful project. Congratulations on shipping it! I would be curious to see ways to do this where the piece of paper had lines so you could visually see on the paper which key was which.

Doing this would be tricky, though. On first thought, an object detection model that detects each key from a perspective, then saves the coordinates, may work. Then you would calculate the Intersection over Union between the finger annd each region. This may only work with a top-down perspective, which might be less practical in real-world use.

In any case, I always miss my keyboard when I am traveling. A project like this feels a lot more practical than the “portable” travel keyboards.

yess I am currently working on that too!

When I was in 3rd grade we learned to type on a “keyboard” that was printed on a sheet of paper. I suppose it’s a novel way to teach kids to type with a budget of zero.

When I was in high school in the 80s, there was a piano class at the high school and the students had non-sounding, non-moving plastic piano keyboards to practice on at their desks. Nowadays, they get 60-key keyboards with headphones.

I know that this is better than nothing, but as I understand, precise fingering is an important part of learning piano.

Is there a risk of kids picking up wrong habits on a paper compared to a (cheap) real one ?

I guess it's always better bad habits than no habits at all.

The context in which this is done is rather important. Also, it might prevent having a ton of lost and worthless keyboards in everyone's attic :D

I have a long password which I don't remember. I can type it on my laptop, but not recall it.

Last holiday season I unexpectedly needed to access my master keyring, but all I had with me was my phone. Which doesn't have a laptop-sized or -shaped keyboard.

No amount of conscious typing helped me remember. As a last-ditch effort, I found squared paper and a pen, drew a full-sized keyboard, and "typed" my password in on that.

It took a couple of attempts, but I did eventually reverse-engineer my own password from my own muscle memory by carefully observing what my hands are doing.

You should write it down. I had such a password that only my fingers knew, not my head. Then I went a couple of months without using it. When I wanted to type the password again, I had completely forgotten it. Without even knowing the first letter, I had no hope of guessing it.

Thanks for the concern, it's stored in a safe place for emergencies :) I just don't often have to use it when on the road.

I'm a pianist and have actually resorted to drawing a keyboard on paper in the past when I've been without a piano for extended periods (yes I'm addicted lol).

Although VR keyboards absolutely won't replace real pianos for obvious reasons they actually do have some use. They are certainly better than nothing and better than drawing a piano on physical paper. The accuracy of finger tracking still needs improvement but that will come and as haptics improves I'm sure a virtual piano one day might start to feel more realistic. Technology needs to catch up.

I've been able to memorise simple pieces with PianoVisions air piano during free play. While playing with songs within the app I've been able to remember some harder works.

Still connecting a midi piano is 100 times better no argument about that.but the virtual piano certainly has its place at the moment and will become stronger as tech improves.

I've previously used PianoVision, and it worked quite well. The concept seems similar, now augmented with VR. I anticipate that VR pianos will evolve to such an extent that they'll replace traditional pianos in many scenarios (like teaching/learning) https://www.meta.com/experiences/5271074762922599/

> they'll replace traditional pianos in many scenarios

I'm not convinced. Totally VR pianos are obviously a bad idea just like totally VR keyboards for typing are rubbish for anything more complex than a wifi password.

And as for learning using VR - well, I'm not convinced by things like pianovision. How do you learn a piece with that? With sheet music I can play the music at my own speed, when and how I want. I can go back and play a measure a few times to experiment with timing. Speed up and slow down, and develop my own ear for how I want it to sound. If I'm feeling cheeky, I'll improvise extra notes in the song, or take some notes out if I'm not feeling it. At an instrument, sheet music is a polite suggestion.

I can imagine playing something like pianovision and getting good at pushing the buttons at the right time like guitar hero. But being a human shaped CD player feels like a very small part of what makes a musician.

Getting good at pushing the buttons is exactly it. Just being able to hit the note that you want to hit on an instrument is a skill

I agree for pianos, because the mechanical feedback is part of what makes playing enjoyable.

I'm not as sure for keyboards in VR! There has been a lot of research on non-invasive brain computer interface (BCI), including predictive systems that guess what you want to type, instead of where you're actually typing (example from 8y ago [1]).

Simpler gesture recognition is already on the market (like this one, from 2020, for the apple watch [2]). And now bigger VR players are investing in the tech [3]. I expect useful brain interface to be integrated in common VR devices in a couple of generations.

[1] Air Keyboard: Mid-Air Text Input Using Wearable EMG Sensors and a Predictive Text Modeland a Predictive Text Model; https://digitalcommons.dartmouth.edu/cgi/viewcontent.cgi?art... [2] https://mudra-band.com/ [3] https://www.androidcentral.com/gaming/virtual-reality/zucker...

I think a melodica is a good comparison and goal.

A melodica is not good for learning piano obviously but is a really interesting musical instrument in and of itself.

Trying to replace the real instrument makes about as much sense as a paper computer keyboard for learning typing. If you really want to learn to type, you obviously are going to "invest" in a real computer keyboard.

If that investment is too much up front then you have already set yourself up for failure so what is the point?

Pianovision encourages connecting a midi piano to the headset. It also uses sheet music and has over 1000 works with finger numbers and sections. Virtual keyboards require haptics to get a lot better, still it is better than nothing at all.

The weight of the keys are an extremely important factor. Pianos and even good digital pianos have hammer action keys.

You have no way of control the volume with this.

I don’t think digital will ever replace a real piano.

Wanted to add:

This isn't to say that something like this can't make its own cool instrument, but as to learning the piano, no matter how good the tech is, its always going to be an imitation of the real thing. Which isn't to say that its useless for learning piano either.

What? Why would they? Digital piano sounds still can't reproduce the richness of dynamics and texture of an acoustic piano. Practising on a digital keyboard and then performing on a real piano is jarring enough, let alone using an instrument with zero haptic feedback. This is pure gimmick.

I'm glad you're finding ways of getting into music inexpensively. But my advice is: haptics are super important in music interfaces! Spend the extra effort to make something that you can satisfyingly press and which can give your fingers some mechanical feedback. Your playing will be sooooo much more rewarding.

yeaa you are right, but those haptics cost money :(

See if you can find – or make! – a cheap one.

Previously:

The pianist who learnt to play on a paper piano https://news.ycombinator.com/item?id=21823936 (December 18, 2019 — 1 points, 0 comments)

It is a video of a guy in 2019 who couldn't afford a piano, so he drew it out on paper and learned that way and became an award winning musician.

Obviously in no way comparable, but you reminded me that many, many years ago, when I was taking typing in high school, I'd sit in class "typing" in mid-air (hands at my side) transcribing what the teacher was saying.

It was a little annoying to people around me, but ended up being very effective.

There was a scene like this in the Talented Mr Ripley (1999).

This is pretty cool work, love the concept. I am curious if there are fast, easy to use and efficient ways to also detect the full hand. Maybe by placing the camera from above would make it easier to process but at the cost of complicating the setup.

I think this idea would work better using 3D printed keys. It's possible to print flexible springs, and filament can be used as a hinge pin. The computer just needs to recognize which keys have moved.

yeah! but main motive was to make it as free as possible. Someone like me who does not have much of money to buy a piano but to play one see this as absolute win with trade offs ofc!

I just made this piano key; the cost of filament is around $0.15: https://files.catbox.moe/kvop4m.webm

So it's in the ballpark of a couple dollars per octave. I agree that's not very competitive with paper.

That's cool, Sounds good. I bet you can sell keyboards with that sound.

This requirement is logical but also.. ouch

> Hand with all fingers.

Where is the file touch_detection_model.h5?

well actually you need to train the model then the file will be generated

Appreciate your work. Happy coding !!

Mayuresh, I'd like to donate a keyboard instrument to you. Please choose between an AKAI MPK Mini and a Hohner Melodica and email me at my nick on gmail with a shipping address.

Also, if you consider this effort much cheaper than a $100 instrument, maybe I could hook you up with a programming job?

I suspect it's less about the music and more about the finger tracking, but to the extent it's about the music - billions of people have smartphones with cameras that would work with this code, but would unlikely be able to afford piano lessons or convince / inspire anyone to offer to purchase them an instrument without first demonstrating aptitude in some area. Definitely not intended at a dig at you and hope you manage to hook Mayuresh up with a keyboard instrument and a job!

Phones have touch screens that are better touch-based MIDI controllers than paper+camera :) I've successfully played songs on phone pianos.

However, neither solution is even close to a tactile instrument.

You are absolutely right, my intention behind it was that you could really play the piano not on a small screen (like a phone) but like an actual sized piano. I didn't want to buy one because they were expensive and I am more like "free is the better" guy. So I decided to make one on paper. This being rather a prototype, I am currently working on designing and mapping keys, improving upon touch recognition. ( also I have mailed you )

I read this morning, I have a half-written response.

Reply on news.ycombinator.com