Keyboard Latency (2017)
danluu.com
danluu.com
Sometimes I try to imagine what things would be like if people were still trying to optimize their stuff as if it were to be ran on a 16mhz machine with a few kb of ram. Then I get back to the 6 electron apps I gotta work with, where latency between a key showing up in the search box and search results appearing gets close to 10s.
Somehow things got wrong somewhere.
Why would that be slow, though? Citrix would only be like that if you were over a high-latency network connection - that's unavoidable. If it is a LAN connection then the Citrix box is overloaded: add more RAM and processor cores (and disable the page-file: disk IO is a big killer of VDI UX).
The problem with the Electron apps mentioned is that it's a reflection of the sad state of affairs that we have crappy UX caused by bad technical decision-making, rather than a crappy UX just because the hardware is underprovisoned.
That's debatable - certainly in many cases it makes a lot of sense: e.g. a corporate environment with hot-desking (i.e. Windows Roaming Profiles), but programmer/dev users need a consistent workspace so they can't use roaming profiles, instead having consistent and persisted VDI environment makes sense.
Another reason is that you can pre-image and snapshot an entire OS environment more easily (and even more easily if you're using a Hyper-V VDI instead of a multi-user Terminal Services or Citrix VDI) which is great for ensuring everyone on a team has the exact same build/dev/test environment. This is being reinvented as "Code Spaces" which GitHub and Visual Studio seem to be really getting behind: https://github.com/features/codespaces and https://github.blog/2021-08-11-githubs-engineering-team-move...
-----
But I'd wager that most devlop-in-Citrix/TermServ situations are in "normal" corporate businesses with "no fun allowed" IT policies, and even if they don't have a hot-desk office policy they probably still don't like the idea of anyone having root/admin access to their own machines. (I'm fortunate to have never been in that situation in my working life, but I've heard plenty of horror stories from friends who did short-term contract dev work on-site for companies like Tesco and the like)
People (mainly EEEs) still do this in the embedded space, where most hardware interaction is done my directly writing to/reading from registers.
Latency for most things is on the order of 100ns - 10us, maybe 1ms or so if you're using a particularly slow serial bus.
The hardware folks gave us magical exponential growth for a straight 30-50 (depending how you count it) run.
That lead to a "we can add more features or we can make it fast" pick one and the business always picks more features. (note: business here just means "non-devs" and can include some devs).
We used that amazing hardware to optimise towards delivery speed over everything else (and while we did/do some horrible things - there is no doubt that a modern dev in 2021 commands processing power and vast libraries that would have been a dream to kid me in the 80's) and when every layer of your stack leaves performance on the table that adds up cumulatively.
Throw in that on the time scales of computers humans are laughably slow and you can get away with a lot of inefficiency.
It's an interesting trade-off - I figure at some point we'll reach the limits of what we actually can do and efficiency will shoot back up the importance chart (in some ways for the hyperscale folks it has for a while, 5% improvement over millions of boxes adds up).
Saying it's "lazy devs not optimising" isn't fair though, they are optimising but in other directions.
Is this really true though? In my estimation the number of features for any given piece of software since the 90s has gone up a little in some cases, down in others (looking at you GNOME), leading to net average of 0.
The software with the most gain in features seem to be high end games and AV editing, which also seem to be performing really well because they aren't typically made in electron.
What does seem to be true is that developers use frameworks like electron because they believe it saves them a lot of time and their time is more valuable than the user's. Well, that and the rampant complexity fetishism.
* Cross-platform distribution. An electron app builds off the extensive work Chrome has done supporting the various operating systems (and chromium has ported it to many others). * Democratizes native application development. It’s not gate kept behind needing C++ developers and makes it easier to have JS developers work on it (+ abundance means it’s cheaper for the business) * Speeds up development. There’s no arguing that ReactJS has been a successfully dominant and powerful JS UI toolkit
Sure, do I wish there were better, leaner UI toolkits out there? Yes. If the goal however is to enable the far greater pool of web developers to also be able to build desktop apps with the same ease as a website, I don’t know that you could do much better than the Electron model (there’s an interesting one that uses the browser engine installed in the OS instead to lower distribution size, which is more of a “different but same” kind of situation” kind of thing). Qt is trying ball compete but it’s really struggling and Rust and other tech stacks are struggling to compete. They have perf, but are lacking in the “enable large swaths of developers to contribute” piece.
That’s not true gatekeeping. Any JS dev is free to learn to write C++ and it's a fairly meritocrital process. I could flip it around and make the same accusation: for decades, "the web" was "gatekeeping" because C++ devs had to learn JS to write apps for the web. Not fair, right?
The C++ (or just not-JS) requirement for desktop software was also not necessarily a bad thing either because writing (correct) C++ or other lower-level code is why native apps are better. They’re written by people that are more likely to understand the cost of memory allocations, cache locality, off-main-thread compute, and the inner working of the OS. The good ones understand how the thread pump works and why just as some things need to happen off the main thread, other things need to happen on it. Web devs are used to everything being on the server (meaning "very, very far away" in terms of latency) but native app devs understood/understand "in memory" vs "on disk" vs "on a different machine." (There are tons of crappy "native devs" that don't get these things and it's painfully obvious when you use those apps.)
I understand why people might be miffed at that, but if technology x comes out tomorrow that lets VC-backed play dough architects build real skyscrapers, are we better off?
The fallacy of thought is assuming that JS developers don't know the importance of those topics when in fact I've met many who do understand all of that (including the inner working of the OS). It may not be the dominant trait, but that would be equally true with C++ developers, you'd just be complaining about all the segfaults & security vulnerabilities are because of the people who don't care to learn "correct" C++. It's the no true Scotsman fallacy. Also, I suspect many of the same memory issues would likely appear if they're in JS land - the JS runtime in browsers seems to be spectacularly good at hiding the costs of things that you might naiively expect to have bad performance in a native language where the compiler doesn't do such tricks. It's everything else the browser is doing for you that's likely the memory consumption (GPU acceleration, accessibility, etc).
I think maybe there's some ways to provide alternate toolkits that are more efficient while retaining JS developers (something like React.Native layered onto Rust+Iced), but ignoring where the market clearly has spoke about development preferences seems unconstructive unless you have some proposal other than "rewrite everything in C++".
> if technology x comes out tomorrow that lets VC-backed play dough architects build real skyscrapers, are we better off?
Depends on what that means, but probably. Faster iteration cycles & lower overheads are very powerful effects to drive down costs. Think SpaceX. You'd probably still regulate it in centralized areas, but in the middle of nowhere, why not let people experiment with what's possible? "Hobbyists" already break the sound barrier racing around on salt flats.
Before you respond, think about it this way. You're complaining about Electron on an article about keyboard latency which is measuring low-level latency issues well before it hits Electron. Clearly it's a challenge for those engineers & that's a very niche problem. They're writing code in C.
https://keminglabs.com/blog/building-a-fast-electron-app-wit...
Right - the question to ask is "what are businesses incentivized to do?" and the answer is "add features".
With some exceptions. Video games, for instance - gamers don't like it when their $60-$80 AAA game runs at 15 FPS on their $3500 rig, so video game developers are actually incentivized to optimize for performance.
If we can change the incentives, then we can influence the outcomes.
One way of changing the incentives is to get users to complain about slow products and use faster ones (although that might be difficult). Another, possibly more feasible way, might be to get federal and state governments to start requiring user-facing software products purchased/contracted to have maximum latencies on a given baseline spec. You can bet that that would cause certain companies to suddenly care about performance.
I still can't find it in me to like the Apple Magic keyboard because I feel like my fingers are smashing into a metal plate hundreds of times a minute. That's a me problem though and others just have better form.
Now I'm curious about the latency of the stuff I've put on teensys in my own custom keyboards/controllers and will need to resist the urge to go shopping for a scope.
Also wondering how much PS/2 vs USB matters.
>>A major source of latency is key travel time. It’s not a coincidence that the quickest keyboard measured also has the shortest key travel distance by a large margin. The video setup I’m using to measure end-to-end latency is a 240 fps camera, which means that frames are 4ms apart. When videoing “normal" keypresses and typing, it takes 4-8 frames for a key to become fully depressed. Most switches will start firing before the key is fully depressed, but the key travel time is still significant and can easily add 10ms of delay (or more, depending on the switch mechanism). Contrast this to the Apple "magic" keyboard measured, where the key travel is so short that it can’t be captured with a 240 fps camera, indicating that the key travel time is < 4ms.
Yeah, I don't get that either.
It should be the time a key makes contact to the time the signal is sent. If I were to design that I would make an external trigger that connects to the solder pads of the key and start the timer on the logic analyser.
I like my keys with a lot of travel and a good thunk when they send the signal.
This would penalize them extensively for doing exactly what I want.
I mean if you want fast keyboards capacitive key caps would be even faster, but I've yet to meet anyone who has replaced their keyboard with a tablet.
Because if we remove travel time by using a capacitive keyboard you end up with something that is objectively a worse keyboard. Replacing the capacitors with strain gauges so you can rest your fingers on the home row without activating them results in an worse keyboard yet again.
Any metric that when optimized produces an objectively worse product is a bad metric and optimizing a bad metric is gaming it.
Prioritising each of those is up to you. Your priorities are different from mine, and my preferences don’t make your keyboard of choice “objectively worse” overall.
I know several people who absolutely loved the butterfly keyboards, and I found them a joy to type on in short bursts. However I, personally, have too heavy of hands/fingers for them and prefer ~80g actuation force so I can completely rest my fingers on the keys without activating them.
So now you have extra latency between key activation, audio activation and haptic activation. You know what gives you all of those for free and ensures they are always timed correctly?
A physical key.
>I know several people who absolutely loved the butterfly keyboards, and I found them a joy to type on in short bursts. However I, personally, have too heavy of hands/fingers for them and prefer ~80g actuation force so I can completely rest my fingers on the keys without activating them.
Activation force and key travel distance have nothing to do with each other. I'm half tempted to build a strain gauge keyboard with zero travel just so people can pay me to see how bad they are.
The less physical feedback a keyboard gives you the more typos you make using it. Arguing that keyboards which encourage you to make more typos are good is as stupid as saying that camouflaged stop signs are as good as the old fashioned red ones.
This isn't an opinion, this is objectively true.
This seems like a tenuous assumption. It completely ignores the main point I have, which is that different users have different preference. That’s why I brought up activation force, because my problem with the magic keyboard wasn’t the feedback, which I absolutely loved, but the activation force of the keys. I and many others have never found the amount of feedback on the Magic keyboard to be inadequate. You are projecting your personal perception that a large amount of key travel distance makes you type more accurately and is therefore desirable(to you), into an axiomatic truth about the nature of keyboards for everyone(i.e. that significant travel distance is needed to make a high quality keyboard and therefore shouldn’t be counted in a metric about the latency of inputs).
The flip side is that there is no expectation of a character appearing until the click happens. So a keyboard with a click which reduced latency by a lot would start showing characters before the click occurred -- and this would be incorrect behavior.
If I'm touch typing on a keyboard with full travel distance, I may have more than one key in motion at the same time, so the latency of each keypress doesn't exactly add.
Additionally, if you're talking gaming keyboards as article goes into, you don't time your "shot" for when you first touch the key. The timing of the shot is relative to when the click happens. You might even milk the key the same way you milk a trigger on a gun. Or if you're in a rapid fire use you flutter back and forth across the threshold with a clickless switch like cherry reds, and you don't use the full travel of the key. Either of these the latency measure in the article doesn't line up with real world use.
Also, it’s a bit weird to say the metric is flawed because it produces a result you don’t like.
Ben Eater has a great video on this:
I had to get to grips with the USB spec when designing my USB scope [1].
Adherence to the USB spec itself doesn't necessarily create human-perceptible latency; HID frames can be transmitted every 1 ms and could be parsed in a negligible number of CPU cycles on an embedded machine with USB-OTG and no operating system.
Like everything, the noticeable lag appears because the application-layer software is going through 15 layers of abstraction rather than talking directly to the input device, and this can become as complex and bloated as it wants.
[1] https://espotek.com/labrador/ (In case you become unable to resist your scope urge :P)
Micro typo alert on the Labrador landing page: "Everything was been" -> "Everything has been" or "Everything was", I guess. :)
I think the only source of polling left is in USB, but I think that's inherent to the USB protocol (someone correct me if I'm wrong here). Without the USB polling I think it would be possible to have key-press-to-USB-packet completely interrupt driven which should make the latency in the keyboard itself negligible.
[0]: I say typically but like you said a lot of implementations are closed source, so who knows. All of the discussions I've seen on matrix scanning use the polling method, and the open source implementations use polling as well (e.g. Keyberon[1]).
[1]: https://github.com/TeXitoi/keyberon-grid/blob/master/src/bin...
AFAIR, the USB poll rate is a function of USB rate, and what the endpoint reports/requests. Low-speed, full-speed and high-speed all have different poll rates.
It’s actually partially why (some?) hard real-time systems eschew interrupts altogether. They introduce a source of non-determinism into the mix as an interrupt handler can stall out non-interrupt performance-sensitive code (or starve it).
https://m.youtube.com/watch?v=wdgULBpRoXk
edit: ah, sorry, I see someone's posted a link to this video already!
The trouble with that project was always that I hate prototyping with BGAs, having to deal with carrier boards sucks, and all the QFP or similar FPGAs were limited to 144 pins, which is just not quite enough for 108 keys given the number of other demands for I/Os, and the number of I/Os you actually get on a 144-lead package.
I wasn't much looking forward to dealing with the USB either (I seem to remember it would require at least a small softcore?) but at least there's plenty of open or vendor implementations there to "borrow".
If anyone wants to rip this off, go right ahead as long as you let me know... this controller was only one piece of a larger idea, and I love it when people do my job for me :)
108 includes 104 plus 4 media keys. I think there are also a few international layouts with more keys than US-standard, but I never got that far.
> I wonder if you could use multiple FPGAs and include a USB hub (soft or real)
That... is a recipe for pain. Complexity of small hardware projects is basically Ackermann in the number of processors you are coordinating.... Ideally in a product you really, really want only one firmware blob to update at a time.
> if you split the ten-key out, that might be enough to do the rest of the keyboard with a 144 pin chip, and the tenkey with a smaller chip.
I don't know if alt-codes would work with this strategy.
Regardless, none of this is worth it just to avoid prototyping with a BGA or dev board. (I'm not even trying to avoid those for production; this is my day job so I'm not that afraid of big BGAs. I just don't want to ever have to solder or probe them, and the gods alone save you if you have to rework a joint!)
Keyboard Latency Connection Price (USD)
chinfai silicone 35 USB FS 17
genius luxemate i200 55 USB 21
logitech k360 60 unifying 25
logitech k120 30 USB 29
easterntimes i500 50 USB FS 31
razer ornata chroma 35 USB FS 59
MS comfort 5000 40 wireless 71
kinesis freestyle 2 30 USB FS 89
apple magic (usb) 15 USB FS 99
das 3 25 USB 101
unicomp model M 30 USB FS 104
hhkb lite 2 20 USB FS 112
pok3r vortex 30 USB FS 129
filco majestouch 30 USB 129
topre type heaven 55 USB FS 149
olkb planck rev 4 40 USB FS 169
kinesis advantage 50 USB FS 319
ergodox 40 USB FS 354
MS natural 4000 20 USB 399
Also, i wish i could vouch for the Unicomp Model M keyboards, however shipping one to Europe would almost cost me as much as the keyboard itself and i'm currently not sure i can exactly afford to splurge like that. Regardless, have a look at their site, they're essentially carrying on with the legacy of IBM Model M keyboards in a way: https://www.pckeyboard.com/
From what i've seen while occasionally visiting their site, they also seem to introduce some new varieties occasionally. From what i've heard, those keyboards aren't exactly ideal for gaming (a regular mechanical keyboard might be better there), but are really good for typing because of their buckling spring design. LinusTechTips also had a video on the Model M keyboards a while ago: https://www.youtube.com/watch?v=D7wmMZmMinM
The Esc key is shifted to the right compared to a standard layout. (Same for the F-keys, to a lesser extent.)
If you're used to the standard Esc position it can be slightly annoying.
For input to screen latency, I think wired keyboard and mouse would be important. For "instant app opening" a recent CPU and enough RAM, and a NVME SSD. But beyond that I'm not sure. Does anybody have any recommendations or tips? Does a 140 Hz or G-Sync monitor help? Are there any pitfalls (I've heard anecdotically that some CPU power saving states can cause micro-stutter)?
Every year I search for a true 10-bit IPS 16:10 (or better) hi-dpi or hi-FPS (ideally both) monitor and every year I’m disappointed. My ZR30W is the oldest part of my PC (Model F keyboard notwithstanding) and has survived maybe four builds.
If you want to avoid microstutters associated with power savings you can adjust the minimum clock the CPU governor will set.
If you want that feeling again you should definitely get a 144hz monitor.
I disagree, but that’s not important. What is important is that there’s a huge difference in the standard deviation of latency (i.e. jitter) and that is - to me - unacceptable.
[1] https://youtu.be/wdgULBpRoXk?t=1953
Even if this conclusion is wrong, that video is a very pleasant and understandable explanation of the USB protocol.
Previous discussion https://news.ycombinator.com/item?id=15485672
I was like - no way this is happening, since Ergodox is using USB cable whereas I used Apple's keyboard before. This seems to confirm my feeling.
Interestingly, I think that it's highlighted the most in Slack which is slow enough bloatware to have a lot of latency already and those extra milliseconds just add up. The simpler the editor the smaller the impact I think.
The issue not mentioned in the article is number of lines on the keyboard matrix used to detect keypresses. Cheaper non-gaming keyboards can only detect up to 4 simultaneous keystrokes. Gaming keyboards can detect up to 6. That may not sound like a significant difference, but if you're moving diagonally (eg. W and D), running (Shift), holding an item (or use) (E) and jumping (space), that is 5 keys which need to be processed. Moving from my older Gaming keyboard to a generic Logitec keyboard, and I was no longer able to run diagonally in FIFA games while doing trick moves. So the non gaming keyboard made me stop playing that game.
Keyboard manufacturers have been designed with n-key rollover as a feature since the late 70s, believe it or not. Keyboard technology has, in many ways, regressed since then. Fortunately, good keyboards are finally being made again—they just need to work on appearance and decent layouts again.
I'm dubious about the accuracy of their lab setup though. From the write up - "The start-of-input was measured by pressing two keys at once -- one key on the keyboard and a button that was also connected to the logic analyzer." that seems like a very week point in what is otherwise a great experiment although they do state "The median jitter was < 1ms" which surprises me.
That's interesting. Has anyone hacked a keyboard? It sounds like a very interesting project.
And I thought human reaction time was 100-200ms+ anyway Or would training on a particular game/movement sequence improve that.
Temporal resolution of human hearing obviously varies with what exactly you measure but I believe it is in a single digit millisecond range for discerning between two clicks in quick succession and a single click.
[0]: https://www.semanticscholar.org/paper/In-the-blink-of-an-eye...
An easy way to cook the results would be to test the Apple peripheral on a Mac and run everything else on your Windows desktop. Frankly, the BSD I/O stack will destroy the one on Windows any day, so it would stand to reason that it might be more of a software bottleneck than a hardware one.
I'm not accusing them of getting anything wrong though, I simply assume they aren't withholding any important testing info.