New acoustic attack steals data from keystrokes with 95% accuracy

Here's a few random papers I read along the way:

https://doi.org/10.1007/s10207-019-00449-8 - SonarSnoop, which uses a phone's speaker to produce ultrasonic audio that can be used to profile the user's interaction (e.g. entering swipe-based passcodes).

https://people.eecs.berkeley.edu/~daw/papers/ssh-use01.pdf - "Timing Analysis of Keystrokes and Timing Attacks on SSH", a paper from 2001 that uses statistical models of keystroke timings to retrieve passwords from encrypted SSH traffic.

https://doi.org/10.1145/1609956.1609959 - "Keyboard acoustic emanations revisited", which uses hidden Markov models and some other English language features to recover text based on classification via cepstrum features.

https://doi.org/10.1145/2660267.2660296 - "Context-free Attacks Using Keyboard Acoustic Emanations" which uses a geometric approach, using time-difference-of-arrival to estimate physical locations probabilistically.

I've always been a bit fascinated by this attack vector and wondered if would get to this point.

Is it more of a physical fingerprint of each key, such that if you swapped keys/springs the model would need to be updated? So it's produced by manufacturing inconsistencies, the way individual typewriters used to be forensically identified?

Or is more each key being identical, but producing a different resonance pattern within the keyboard/laptop due to the shape of all of the matter surrounding it? If you move the keyboard in the room, do you have to re-train the model?

I also wonder how much it varies depending on how hard you press each key -- not at all or a great deal? And what about by keyboard -- when you compare thin MacBook keys with an external full-height keyboard, is one easier/harder to recognize each key on than the other?

I'm pretty sure Zoom does this by default as part of its noise cancellation (it's potentially even easier since you can use keydown events to help identify, not just the audio stream).

So as long as basic default noise cancellation is on, that would at least prevent this over regular videoconferencing. And because of this, I'm having a hard time thinking of when else this would be a realistic threat, where the attacker wouldn't already have enough physical access to either install a regular keylogger or else a hidden camera.

https://github.com/ggerganov/kbd-audio

A couple years ago for a weekend project I made a simple "audio-mnist" dataset from handwritten digit audio recordings. I never got past a few days worth of work, but open-sourcing it has been on my mind for a minute. This post kicked me into action. Getting some more data, basic CNN examples, etc. could provide a nice starting point for a lot of research and tools.

There is still separate code I'd have to find and make intelligible to create the recordings and split the audio.

Anyway, in case anyone finds part of this process interesting or useful.

In 2005 ACM's CCS Zhuang, Zhou and Tygar presented Keyboard Acoustic Emanations Revisited [1]

    We examine the problem of keyboard acoustic emanations. We
    present a novel attack taking as input a 10-minute sound recording
    of a user typing English text using a keyboard, and then recovering 
    up to 96% of typed characters. There is no need for a labeled
    training recording. Moreover the recognizer bootstrapped this way
    can even recognize random text such as passwords: In our experiments, 
    90% of 5-character random passwords using only letters can
    be generated in fewer than 20 attempts by an adversary; 80% of 10-
    character passwords can be generated in fewer than 75 attempts.
    Our attack uses the statistical constraints of the underlying content, 
    English language, to reconstruct text from sound recordings
    without any labeled training data. The attack uses a combination
    of standard machine learning and speech recognition techniques,
    including cepstrum features, Hidden Markov Models, linear classification, 
    and feedback-based incremental learning

    We show that PC keyboards, notebook keyboards, telephone 
    and ATM pads are vulnerable to attacks based on
    differentiating the sound emanated by different keys. Our
    attack employs a neural network to recognize the key being 
    pressed. We also investigate why different keys produce
    different sounds and provide hints for the design of homophonic 
    keyboards that would be resistant to this type of attack.

[2] https://ieeexplore.ieee.org/document/1301311

Note that the testing data in the confusion matrix appears to have a uniformish distribution of each key being pressed. I suspect this data was not generated by someone actually typing because you would rarely see numbers and rare letters. It is possible these were simply pressed one at a time rather than in a series of rapid presses.

My guess is this approach uses the mic to identify where the sound of the key press was coming from rather than what each key press sounds like. Which does not invalidate the results but may make it seem less magical. Tbh it’s probably much worse this way because such a model could probably generalize very well across all keyboards and typing styles.

It would have its own set of problems: not two people using it at once, eavesdropping would be really easy… but it’d have its own set of interesting applications

https://youtu.be/a9Gz7Bg07u8

This attack is about as realistic as the film: a parallel universe where million to one chances happen nine times out of ten.

I suspected that the famously terrible Treasury Direct website with its on-screen keyboard was a half-assed attempt to prevent this sort of attack.

It would be nice to try to tokenize the strokes and then try to assign labels probabilistically.

Do we need something similar for microphones too?

Suck it.