Markov Keyboard: keyboard layout that changes by Markov frequency (2019) | Hacker News Reader

Markov Keyboard: keyboard layout that changes by Markov frequency (2019)

github.com

167 pointsdr_kiszonka105 comments

Nice. Some thoughts.

Look at PPM. Your prediction model would work better with personalised data. PPM is efficient (nb. I see you are using python - look at this https://github.com/willwade/pylm - although be warned - i think my code is not quite right..)

Layout shifting for finger movement - well its great if you didnt have to look. The time for visual processing the letters adds a significant lag (its why typical word prediction isnt used that much and when it is - not over 3 predictions (I have papers on this if you are interested). But its not all bad..

Switch users who need next letter prediction this could dramatically support their rate of input. (view https://youtu.be/Bhj5vs9P5cw?si=VnytfH_vdEUWuLok&t=73 - now note how the keyboard blocks the scan up. But imagine if it just scanned each letter first by next most likely - or heck - like this repo - actually changes button position and kept the scan pattern the same. It would be a ton more efficient)

(and a bit of a rabbit hole.. What if keys had word predictions on them? This is basically the end result of ACE-LP: https://discovery.dundee.ac.uk/en/publications/ace-lp-augmen...)

Yes! I have been wanting this forever. Keys are in too predictable a position

Exactly, a key should never be in the same place twice in a row- that's just wasted movement that could be better optimized. Letters the computer predicts you probably won't use should automatically disappear, and be replaced with numerous copies of others based on the things it thinks you will type.

We should extend this concept of constant automatic optimization to all aspects of everyday life- for example your workplace location should physically relocate each day to an optimal location based on where each person coming in that day lives. An algorithm should tell you where to put away the dishes in your kitchen based on a constantly changing optimization algorithm, so that your dinner plates are in a different cabinet each day. Language itself should be radically redesigned daily to keep it optimal, with changelogs pushed out to be learned and memorized each morning before communicating with anyone.

A good part is that it will do nothing to improve security and will actively harm peoples' ability to type things correctly, and the best part is when it makes its way into scientific literature and algorithms. Good luck reproducing that which was entered incorrectly!

Presumably if something is sufficiently impossible to interact with in any capacity both for the intended audience and nefarious actors, it is also "secure." For example, if your computer keyboard layout is scrambled through a one time pad, where no copies exist except the internal one, inputting non-random data of any kind would be provably impossible.

Air gapping is a legitimate security technique. However, in this case the air gap would be between the ears of whoever designed it.

There were, for awhile, some security systems had on-screen keyboards that would change layout on every key press.

I think that this is an attempt to stop the “clean key” problem. Security system keypads (especially outdoor ones) tend not to be cleaned, so as time passes, it is easy to spot the dirty keys. Dirty keys are not being pressed and are therefore not in the passcode.

So look for the clean keys and try combinations from there. In a 4 digit (0-9) keypad, knowing the clean numbers drops the possible codes from 9999 to 24 (if my early morning math holds up).

Also helps the issue of someone looking over the shoulder of a valid person. Chances are they are just seeing the position and not the character pressed. So the keyboard changes and you actually need to know the character not just the old position.

It would be 24 if all digits are distinct

It actually drops from 10^4=10000 to 4^4=256 combinations

If they aren't distinct, you wouldn't have 4 clean buttons, but just 3 - in which case we also know the repeating digit repeats exactly once and we get 12x3 (36) possible combinations. With two clean buttons, it's 6 (if both repeat) + 4 (if only one repeats) = 10 and if there's 1 that's just one, and a terrible password.

With 2 clean buttons, there are 4x2 ways for only 1 to repeat, giving 14 combinations in total.

I encountered a case of this in college, where there were four clean digits - a tough task to be sure! But fortunately the digits happened to be the same set of digits that comprised the room number. It took two guesses, because there was a twist - the combination was the room number, backwards.

I guess you would be able to count the number of clean keys and thus know both the number of distinct digits and the digits (but not the order nor which digit that's repeated)

The number that is repeated is likely to be dirtier than the numbers that are not so you get that information too.

Interesting. This would also stop keypress extraction via analyzing audio.

Hmm... It may still be vulnerable if:

1. You have lots of spy-data samples that reveal which physical key is pressed (perhaps they sound different) and the precise timing of those strikes, but you don't know what scrambled numbers were actually being shown. (And it's always the same code.)

2. The trick is that users take longer to press a number when it's displayed far away from its "normal" position, because they had to seek longer to find it.

3. This means you can infer the true numbers based on how quickly or slowly presses happen versus which physical key is struck.

For a simple example, assume a two-digit code where there are nine keys. If the fastest first press is always the top left corner, and the fastest second press is always the middle, we can guess the code is either 15 or a 75, depending on if the user is accustomed to phones or keyboard numpads.

P.S.: On reflection, I could probably have shortened all that by describing it as a "timing attack" [0] except in meat-space.

One mitigation might be to get the user to enter digits at a consistent pace, by forcing a delay between showing the random layout versus accepting a button press. There would need to be some penalty for early presses, to keep lazy users from just tapping the desired button repeatedly until it became active.

Also spy robots with thermal imaging eyeballs.

This would be an interesting one to integrate into password entry forms... although you'd need to show the randomised keyboard layout on screen.

Or have a keyboard with oled or e-ink keys, like the Optimus Maximus [0] promised to deliver. It's kinda weird that nobody else seems to have picked up on this concept since then. Probably just impractical or too expensive.

I read that its patents expired in 2016; around 2015 there was a concept for an e-ink button keyboard, but that site is now a plain gambling ad. There's also https://www.nemeio.com/ that still works, but its buttons look like sunken screens under plastic domes.

[0] https://en.wikipedia.org/wiki/Optimus_Maximus_keyboard

It’s still a relatively common thing for pin-coded door/gate security.

My bank does this, you can see it at https://www.ing.com.au/securebanking/

It also takes ~5s to render the page on my PC, no idea WTF it's doing during that time, probably mining crypto.

Hmm actually, it doesn't rearrange on every digit, it's a static but random layout every time you go to that page.

A number of countries use this when giving your pin for a credit card or similar (I've noticed it in both Greece and India).

I can't help but feel like it's less secure than the default layout - I'm quite good at hiding my PIN and typing quickly, but when the positions of the numbers are randomised, I feel like I practically end up saying my PIN out loud as I try and remember it.

Had this at an ATM recently, and it took a couple of tries at my PIN before I looked at the keypad and realized what was going on. One more wrong PIN and I could have lost my card.

I occasionally still get this in certain petrol stations. Always catches me off guard.

I have seen this with some of the card swiping machines in India.

One way to avoid this element of surprise, for those who do not enjoy surprises, could be to have only a single key whose action upon clicking would simply be the most probable next action.

This project brilliantly points out that we are limited and shaped by how keys and layouts are presented to us.

What I don’t understand is why not go all the way. Why not also change the concept of what’s needed to enter a character? It would make sense that keys aren’t the ideal affordance to express a key every time.

Use a volume slider for a letter, use radio buttons, on/off toggles. What if I could draw the letter on my trackpad?

I hope this opens the door for a thriving ecosystem of expansions.

> What if I could draw the letter on my trackpad?

I have been proposing to adapt Shorthand* to computer tactile input many times. Touchpads and touchscreens can use it. It seems faster and easier for both the computer and the person than having to recognize slow handwriting.

Secretaries used this for many decades, because it allowed them to record speech in real-time.

There is always a dismissive comment about how this is not a good idea for one reason or another, but I still think it would be a great feature to have.

* https://en.wikipedia.org/wiki/Shorthand

> What if I could draw the letter on my trackpad?

The Apple Newton did that!

https://en.wikipedia.org/wiki/Apple_Newton#Notes

Also the palm, they called it graffiti. https://en.wikipedia.org/wiki/Graffiti_%28Palm_OS%29

(Google's gboard offers a handwriting keyboard, but it's on the touch screen and doesn't count. Also, it's not graffiti, so a bit more error prone. It does recognize joined-up handwriting though.)

There's also some keyboards on f-droid for braille text input and morse code input.

There were a couple of aftermarket systems for the Psion as well. Normally the keyboard was better, but writing on the screen turned out to be easier in a bumpy taxi.

The Apple Watch can do it today.

(In iMessage, as one easy example app, there's an entry box to send a new message. If you tap into that and get a keyboard click the keyboard icon in the bottom corner and there's a "moving finger" icon you can tap to get a trackpad to draw letters on. Older and smaller Watches won't even have the keyboard and will just have the trackpad.)

That feels like Siri dictation, minus Siri, and minus the dictation

Heard of dasher ? https://www.inference.org.uk/dasher/

I had the idea this week to use the Rocksmith USB adapter to develop a stenography system that allows me to control my computer using my guitars with chords, notes, and riffs.

There's a Rocksmith mod that allows you to control the menu using the guitar. Also, the Rocksmith USB cable is simply an ADC (with lower latency than most)

We're also limited and shaped by keyboards. Why only one letter per finger? The datahand does five!

Why are key switches only press?

What about analog partial key press? There are Hall effect keyboard switches that do that, I want to try some.

What about finger twist for input? I don't think anything exists? Would that be a volume knob on a key switch?

> What about analog partial key press? There are Hall effect keyboard switches that do that, I want to try some.

That sounds fun, now I really want to try out a keyboard where half press is lowercase and full press is UPPERCASE! No more pinky strain from shift presses.

Some pipe organs have keyboards that support that. I can't figure out when it started, but my guess is the 1920s. No hall effect, they had mechanical contacts. (I've never got to play one and always wondered if you could feel when the two positions were hit or if you had to guess and rely on sound - organ pipes often have a long delays between when you press the key and when the speak)

A simple solution that would work on any keyboard (but will be much easier to do on a keyboard that supports custom layout and layers, e.g Ergodox) would be to send small letter of key is pressed under 0.5s (to be tested what value feels best) and capitalized if over this value. After a while it might feel natural to press quickly for a small letter and slightly longer for capitalized

The ZSA firmware fork of QMK does this! I don't have it enabled on my moonlander, but many of my friends do.

This is true. We’ve been so formatted by big keyboard that I never considered just the tip of my fingers. I hope you’ll make good use of our opposable thumbs in this new keyboard paradigm

I agree. I blew out my left arm for a year from too much typing, so I switched to kinesis style keyboards so I could put modifier keys under my thumbs. I dunno if your suggestion was meant to be serious, but it's still great.

It wasn’t, but the reason I like to explore absurd looking ideas is because of the serendipity that lateral thinking opens up. Thanks for exploring that!

What would be a cool use case for thumb controls here? Is this something you could do with those gamer keypads?

I was pressing control and alt and shift with my pinky fingers, and that's how I blew out my arm.

I switched to a Kinesis Advantage keyboard and now I have control, alt, hyper, super, backspace, and shift under my left thumb. My right thumb has control, alt, hyper, super, enter, and space.

I don't know anything about gamer keypads, is there one you suggest I investigate?

I’d start with this https://en.m.wikipedia.org/wiki/Gaming_keypad

The CharaChorder [1] is one attempt to give a completely different shape to input methods

[1] : https://www.charachorder.com

Something in this vein has actually been made quite some time ago: https://www.bltt.org/software/dasher/

I'd put some random keys on the bottom of my seat or wired into the TV remote or the doorbell button of my neighbours. The ordered shape and regular size, position of keys is too much of a burden to mee, constricts my wondering creative mind too much.

I wrote a second blog post about this project weeks after the first release: https://www.scannedinavian.com/markov-keyboard-the-rabbit-ho...

Hi shae!

Took me 1 year to rewire my brain to switch from Qwerty to Dvorak (I struggled with from RSI for nearly 6 years) and nearly 2 years to gain full speed. It is hard, very hard for the brain. It is mentally tiring to rewire your brain like this and you will do your job at a much worse efficiency (forget pair programming for a while). You need new keyboard stickers also. Great job at making this idea but it is not practical.

> You need new keyboard stickers also.

You absolutely do not and it even hampers your ability to quickly learn to type efficiently if you rely on key legends or layout references.

I switched to Dvorak 20 years ago and I love it. It only took me about 2 weeks to learn and to pass my qwerty typing speed. I could actually touch type on either keyboard for the first year or two after I switched. I recommend not using stickers or dvorak keyboards at all, it helps you learn faster and your family won't get pissy because they can't use your computer anymore.

Also, it is not difficult to use other peoples computers. You can switch keyboards pretty easy these days on windows and mac. In fact, my very first computer (Apple IIc) had a mechanical switch to flip between qwerty and dvorak.

And there's also the problem of using computers that aren't yours. I haven't tried (and I will never), but I imagine switching between layouts is at least very inconvenient.

Dvorak is supported by almost every computing device under the sun today. Old Apple models (I want to say the IIc?) even had a hardware button to switch layouts. Switching between Dvorak and Qwerty is usually an easy to find User setting. (If doing it on someone else's user account, you just have to remember to be kind and switch it back when done.)

As a Colemak touch typer I envy that some days. Colemak is available easily everywhere but Windows. macOS, iOS, Android, and Linux all have it similarly out of the box with Dvorak, but Windows it still requires an install and that install still requires Admin approval because keyboard layouts are needed by kernel-level drivers. But I knew that when I switched, took a few years to realize there's no shame in hunt-and-pecking on QWERTY when using someone else's machine.

(I was also privileged when I switched to know that I didn't need to use shared machines. I was far enough along in grad school where I was allowed/encouraged to do everything on a personal laptop, and the few remaining "lab classes" with shared machines all allowed Remote Desktop. That took advantage of things like if you remote desktop back to one of your own machines, your remote machine still responds to your chosen layout once you've logged in. Just have to hunt and peck long enough to type your machine's address, username, and password.)

I switched when I was a teenager and have Thoughts (TM) about typing Dvorak, though I'm locked in at this point. The most painful experiences typing with it are:

1. Using a shared machine for something important. I've had to hand write and then hunt-and-peck essays on shared machines when I was still in college.

2. Typing something on a friend's computer when I'm inebriated. Not a huge problem nowadays with phones and being older, but in college trying to change the playlist when I was inebriated on a friend's QWERTY layout was... tough to put it mildly.

3. Piloting the Media PC with my partner. She types QWERTY and I don't. I keep Dvorak around as a second layout and switch if I need to do a lot of typing on the Media PC.

There's lots of both pros and cons of using Dvorak in my experience but at this point I'm so locked in that I'm not going to switch. But in my adult life, I never have to use a shared machine either so I'm always a few feet away from a Dvorak keyboard.

It's also interesting how #2 is shifting in recent years. These days it is more likely to be someone handing you their phone to change a Spotify playlist and while I touch type Colemak, I tend to swipe type QWERTY on phone keyboards, so it's not as tough now than when it was someone's laptop/desktop running the playlist.

Also, Apple Music's SharePlay is such an interesting modern approach too, where multiple people can control the same playlist from multiple devices.

Yeah, this is why I picked Dvorak: it seems there are better layouts but Dvorak picks up the greater part of available improvement, and it's more generally available.

I switched to Dvorak ±8 years ago, and I can still type qwerty almost as well as I could back then. I could imagine it being a problem if you're using other people's machines on a daily basis though.

This entire discussion reads like perfect satire because I still can't decide whether we are being serious or not

This mixed with LED keycaps that change to reflect the current assigned letter would be awesome.

For hunt and peck typists.

Seems like eink would be better, but the smallest I can find is about 3cm so much too big. If you can get them smaller though...

Nice! I wrote about something similar for rectangular layouts: https://maxhalford.github.io/blog/dynamic-on-screen-keyboard...

thats wonderful. id like that to not change the order of the letters - but change the highlight order. Do a round 1 of frequency order first (just do first say 6 letters) then do a round 2 which is standard order..

i probably am not making much sense. Look at where I'm coming from in the world of Assistive Tech - https://docs.acecentre.org.uk/products/echo (go to around 5 min mark in the vide)

How about a backlit keyboard, and if the next letter is probably a vowel, only the vowels are lit?

I'd be quite interested in trying that. Maybe as a learning aid. But if you get your typing speed up, I can see it being visually very noisy.

Well I knocked something together thanks really to @Lemaxoxo for his blog post inspiring me to do this.

https://scanningmvp.netlify.app/ - it’s more for my needs of an AT keyboard for scanning than your backlit one but you get the idea.

I learned recently that there are way, way more alt-layouts than the ones you've heard of: Dvorak & Colemak.

Not only that, people just change them to suit their needs! They use things like Hands Down [0] as a guide, and make up their own layout.

Wild.

[0]: https://sites.google.com/alanreiser.com/handsdown

Thank you for the link! That looks fascinating. I have instantly added it to my "keyboard" bookmarks folder!

I will promptly forget it exists and keep typing on the random layout I learned to touch type on 20 years ago, all the while suffering from terrible finger acrobatics and wishing for a better layout every single day.

I've made my own layouts[0][1] and I'd say the many variants of Hands Down are probably the best example of a layout that's been developed in a thoughtful manner.

[0]: https://www.jonashietala.se/series/t-34/

[1]: https://www.jonashietala.se/blog/2024/11/26/the_current_cybe...

Would you say they are hands down the best?

(I do believe that the best for you is a custom layout.)

Very nice website with many solid principles! Though one big flaw in all the designs I've seen is that they rely on bad data - text corpus that is the output of human editing efforts, not the input.

So all those "0.00001%" stats aren't as precise as they seem, and you still see the frequently used backspace/enter in the awful pinky position despite the official philosophy of "Hands Down is easy on the pinkies"

Have you seen any design that is based on actual human input?

As far as I'm aware hands down layouts were primarily designed to be used on a split ergonomic keyboard with thumb cluster, the space and enter lives on the thumb cluster.

Even those better thumb variants don't seem to statistically take that into account (judging from the description/heatmaps/data sources), but then it also presents the "slab" variant...

For those who type on German keyboards or in German, there are a number of layouts that have evolved around the Neo 2 layout [1]. The main driving force is ergonomics, but since not everyone works the same way, many variants have emerged (and still do) [2]. Interesting is the craftsmanship, including customized programs to optimize the layouts [3].

[1] https://www.neo-layout.org/Layouts/

[2] https://maximilian-schillinger.de/keyboard-layouts-neo-adnw-...

[3] https://hg.sr.ht/~arnebab/evolve-keyboard-layout

German layouts with [ ] { } in such horrible positions are killing all joy when programming, especially in curly brace heavy languages.

Changed to US qwerty 30 years ago, never looking back.

You don't save much. {} still requires a modifier key. You get [] for "free".

There is a whole usability universe between shift and AltGr, the former being accesible with the other hand.

I always changed to ANSI layout, independent of the layout of the actual keys. Now I carry a HHKB with me, and connect through BT to whatever computer I want to work on. No matter what human language I write in (I do some) I prefer using accidents like tildes and umlauts with modifier keys.

And there are modal editors, who switch keylaouts depending on the need of the minute.. debug/edit/search

https://keyboard-design.com/internet-letter-layout-db.html

site is a bit awkward to browse though.

If you're interested, look at the layout doc: https://bit.ly/layout-doc-v2

Yeah when I was a 20 or so and just started working I taught myself Dvorak for a while, but realized how bad it was for coding in C. So I made my own C language one as well.

Of course you still have to type text and at the time you couldn't easily switch between different layouts, so after a time I got bored of it and gave up.

Keyboards aside, I find it really frustrating when something I use frequently - such as the toolbars in excel - changes without warning and breaks my muscle memory.

This is close to an idea I had in the past.

I bought a keyboard last year that has a small embedded CPU that I can reprogram with QMK [1]. I designed an objective/score function for key placement, based on frequency of letters (and groups of 2 and 3 letters) and finger movement (in the case of strings of letters). Each finger was weighted by how much I didn't want to move it haha. I used this function to optimize my "perfect" keyboard and reprogramed the keyboard to match it. I'm still sometimes trying to learn it to this day, I don't use it often enough. My biggest problem with it is that it is an ortholinear keyboard... Not confortable enough. Anyways.

I had plans in the future to make a small keylogger on my PC to follow my ACTUAL usage of keys, and run the optimization again automatically. Then the software could message me what would be changed and upload the layout on the keyboard every X month (not too often, y'know).

I'm waiting to get another keyboard for this, probably a split keyboard.

[1] https://docs.qmk.fm/

This is not a bad idea. It's not dissimilar from an autocomplete engine. If reset per word and using a static probability table it will work its way into muscle memory. In practice fingers will only need to leave the home row for some initial letters.

What letter is a good start if you do a reset at every word boundary? Maybe whatever letter is most common?

E I guess?

I'm still looking for a way to produce quality keycaps at home.

I too am looking. I started drawing on a napkin how it would work. Basically injection mold the letter and stem only in one color, then put that in a different mold and injection mold the rest of the key around that. Seem like it should work, but I haven't done enough injection molding to know how feasible it might be (letters don't seem to bad, but 5/% probably needs a 4 part mold or something and I can't imagine keeping all the parts together). Worse, there is a lot of tiny machining operations needed and so for a one off my cost per key gets to be $15 each (mostly on tiny bits needed that will thus break often).

3d resin printers might be a good option, my background leans to injection molding so I didn't think of it until now. Here you would print two pieces and then snap them together.

Both of the above have the letter molded through the key and so whatever chemicals are in my fingers won't destroy the lettering after a year of use. Though plastic selection is also important otherwise my fingernails will destroy the key anyway in a short time. Not everyone has fingers/skin chemistry like mine and so some won't understand why I care while others are all in with me.

ah, I have been looking for this all my life, thank you! ;-)

(though really what I want is dabbrev/pabbrev to sort predictions by markov order at least 2 or even a tiny recurrent network)

I’ve backed this product, the Flux Keyboard, on Kickstarter - it allows for the type of dynamic changes that the author is describing in case others are interested:

https://www.kickstarter.com/projects/fluxkeyboard/flux-keybo...

Similar setup to the Optimus Popularis around 2011

https://www.tomshardware.com/news/Optimus-Keyboard-LCD-Scree...

Before that they had some designs with little displays in each key, instead of a big display behind all of them

Fun concept. I had a thought about having it optimize in a way where left and right hands alternate for increased speed

Could a LLM tailor the keyboard to intent ... as in it shifts keys around to help you write what it thinks you want to write? like a sentence starts with a if then the left keyboard becomes permutations of an future if clause on key,the right becomes else statements..

A real shame this doesn't seem to remap arrow keys, or it could come in handy for my next HATETRIS [1] run!

[1] https://qntm.org/files/hatetris/hatetris.html

I switch between English and German a lot. I wonder what that does to the algorithm.

There is an old joke that seems approproate:

https://old.reddit.com/r/Jokes/comments/2o4rkq/english_to_be...

> (…) In the first year, "s" will replace the soft "c". Sertainly, this will make the sivil servants jump with joy. The hard "c" will be dropped in favour of "k". This should klear up konfusion, and keyboards kan have one less letter. (…)

That joke is many decades older than the creation of the european union: http://www.i18nguy.com/twain.html (probably not written by Mark Twain, but seemingly first appeared when he was still around?)

Very interesting, I wonder if you could incorporate text prediction so that you can write a sentence in just one key (aside from space).

Repeatedly pressing the home row keys almost always turned into word end sequences like erere, I assume from words like there/were/here. I'm not sure how to go further with single character frequencies.

Too bad this github has latest changes 5 years ago, I wonder what they could update with LLMs, etc.

My guess would be not much. LLMs are pretty useless concerning anything novel. Maybe an LLM could update the docstrings?

I'm assuming they meant replacing the markov model with a modern language model, not having an LLM magically improve the repo.

That sounds like it'd give you no hope at all of ever learning muscle memory for it.

Reply on news.ycombinator.com