Typometer: A tool to measure and analyze the visual latency of text editors
pavelfatin.com
pavelfatin.com
It's unfortunate that software seems to feel slower the newer it is, including developer tools such as text editors/IDEs. I suspect this is because most people - including younger developers - have never seen how fast things can be, but are instead constantly subjected to delays in using applications to the extent that they think it is an absolutely normal speed[1], and then propagate that notion to the software they themselves create.
Also related: https://danluu.com/input-lag/
[1] For example, everything that uses UWP in Windows. Someone who had only ever used the Settings app in Windows 10, or (even worse) the Calculator, might not realise how absurdly slow they are in comparison to the previous versions' where they would open nearly instantaneously.
Scroll a bit down for Windows with power saving, Linux (section 3.4), and VirtualBox (3.5).
Terminal Emacs over ssh is, well, just like anything else in a terminal over ssh. Can't say I notice the latency unless the datacenter is on the other side of the country.
That, plus the ability to rejoin a session even from a different IP, makes working over SSH doable even from airplane Wi-Fi.
I like the UI simplicity of VSCode in contrast to native editors such as IntelliJ and Visual Studio, which I do use when I really need the features. I am interested in efforts like Onivim 2, which seek to combine these advantages.
For me at least.
I've found that I get perfectly passable syntax highlighting with vim, and haven't needed anything else on my current install. If I did use an editor other than vim, I would probably use VSCodium because open build, and I do really like the visual style.
The latest was I typed
...nameOfArrayInOtherFile
import nameOfArrayInOtherFile from './tests/name-of-array-in-other-file.js';
Another thing I've seen VSCode do is give me library specific warnings. Maybe that's common now-a-days in other editors but I hadn't seen it before. I'd seen language warnings but not library warnings.
Annoyance aside I seriously worry about the accessibility of the windows lock screen. Through a weird display setup I sometimes find myself trying to log in blind and I just can't do it. There seems to be no way to reliably focus the password field, and for whatever insane reason it's not just always in focus
If you try running a screen reader, such as Narrator (built into Windows; edit: turn it on and off with Ctrl+Win+Enter, even on the logon screen), you should find that even if the focus somehow gets away from the Password field, you can get back there with the help of speech output. Logging in blind doesn't have to mean logging in with no means of orientation.
(Disclosure: I'm a developer at Microsoft on the Windows accessibility team, working on Narrator among other things.)
The reason it works like this is that the user is forced to use the software. It should be the other way around. If people are forced to use something developers should do what they can to not make the experience incredibly annoying.
Sure, there can be situations where you need to cut corners but this is the log in screen.
After logging in I mistype an application name in the search box and wait for the dialog to pop up that says "Best matches" with an empty list under it. It can't even find "tunderbird"
If I type a single letter it shows "search teh web for "a".
But it does show Atom! Woah! Technology at work!
I get to chose, I can press the down arrow and select "apps" then press enter twice OR I can press the down arrow twice and press enter! Becareful! Dont press the arrow to fast or it jumps back to searching the web for "a".
No one put any thought into this process.
You're making an uncharitable assumption about real people who work full-time on this stuff. I can assure you that they do put a lot of thought into their work. No, I never worked on the Start menu or search myself, but before we started working from home, I was in the same building with the folks that do.
(The older versions of Windows had a native dialog box: http://www.guidebookgallery.org/pics/gui/startupshutdown/log... )
I don't use them.
I use Open-Shell (formerly Classic Shell). Before that I used Launchy (which is another crazy-fast alternative) tied to my CapsLock key.
I've also had tabs on my file Explorer windows, since Win 7 (thanks to QTTabBars). I'm commonly asked about that after I give remote desktop presentations.
It's just glaringly obvious that nobody has bothered for even a second to think about how to solve the problem (or indeed, just copied the solution that already exists elsewhere).
The start menu on the other hand I gave up on several years ago. I don't know what's in it anymore, and I don't care, because nothing could be worth the >5s wait for it to open.
This poorly named option doesn't disable locking the screen, it just fixes it to not eat your first few keystrokes when you start typing your password.
Is it even possible to do it through system settings alone, even if obscure ones? I use AutoHotkey to do Caps Lock -> Ctrl remap, and it's one of the first programs I install on every new system.
Thanks for the AHK tip. The programs I usually find look like hastily-constructed malware on a dodgy sourceforge link & always make me feel real :-| about installing them on a fresh box.
Wow, how the time flies!
I was completely blown away by the responsiveness of the controls. The control feedback was instant, frames were rock-solid, and there were no loading screens at all.
When I put down the game, I was just tremendously sad. I'd forgotten what instant feedback felt like.
You just can't get that responsiveness anymore.
It's that old consoles (up until the Sega Saturn, Nintendo 64, Sony Playstation era) didn't draw to frame buffers. They didn't have enough RAM to have a real frame buffer.
You configured the graphics chip what you wanted it to draw, and as the scanline scanned across the screen, it outputted the color that was supposed to be output depending on the background/sprite/palette settings. You could change where the sprite is halfway across the scanline, and it might immediately, in the middle of drawing a scanline, output a different color. There was no delays anywhere in the system, only the propagation delay of electricity in wires and transistors. The various HW registers that controlled what and where the sprites were were connected via a small pile of non-clocked digital logic gates to a DAC to the analog output pins of the graphics chip, which was connected to the electron guns directly if you used component video or via a simple, no-latency analog circuit if you had composite video.
These days, you program the graphics card to draw what you want it to draw. Once you've programmed everything for it to draw, it draws that into the backbuffer. Once the current frame has been sent to the display, the backbuffer is swapped with the frontbuffer. When the next frame begin is sent to the display, your changes finally go out across the wire. Depending on the display (see especially motion adapting TV screens) there might be more delay. Then whatever changes you make are finally displayed on the screen.
It's kinda weird. Old video game consoles (up until the SNES and Genesis) had extremely low latency. And that's been gone for 25 years. Not only is it gone, but it will likely be gone forever- we don't even make the technology anymore to show the new generation what it was like. On the one hand, the new technology is "better"; there's no way to do today's advanced graphics without a deep drawing pipeline that outputs to a frame buffer. But it's also somehow worse. We can make it less bad with technology like 144Hz and Freesync, but the old era is gone.
I started on the Apple II and i'ts been downhill since then. If I remember correctly the Apple II has one of the smallest latency between touching a key and seeing the result on the screen.
The N64 had a unified memory architecture, so the frame buffer was just a region of memory within the unified system RAM that was drawn to. The Z buffer was the same way. (assuming the programmer chose to enable to the Z buffer) It was still a frame buffer though.
So for a given set of timings, you'd always expect the CRT to be faster, because it represents the absolute minimum time-of-flight delay, assuming it is driven by a direct RAMDAC and not by a conversion box buffering an entire frame or significant portions.
Also, you can "move" electron beams at extremely high speeds. Even in a magnetic deflection CRT like the GDM-FW900 the electron beam can move at more than 80 km/s[1]. In some electrostatic deflection systems the speed of light is exceeded: they can draw a dot moving across the face of the CRT that moves at a higher speed than c. This is possible because "the beam" is a fictional object.
[1] 2304 dots across 482 mm with a pixel clock of around 384 MHz means these 2304 dots are covered in about 6 µs; 482 mm/6 µs = 80.3 km/s.
I was curious so I did some tests:
It is on par with "the fastest terminal emulator in existence". :)
Newer tools only come as alternatives to faster tools that continue to be usable at any time.
For instance I choose to use VSCode and consciously weight it down with plugins and extra linters, because the trade-off is fine for me.
But I know vim is only a click away, and if I wanted sheer speed I’d do it there. And I actually do use it on a day to day basis, it’s just not my primary editor.
Also, it is worth noting, that the same person doing this research is the one working for JetBrains on optimizing editor latency: https://blog.jetbrains.com/clion/2017/01/clion-starts-2017-1...
So there are companies taking the editor latency issue seriously and that are close to fixing it.
Debouncing in software is one of those things that 99 % of developers get wrong, and is something even hardware manufacturers get wrong all the time.
A lot of hardware debounces in a dumb and naive way: On the first state change, it waits 5-10 ms and sample the switch again to figure out if it was pressed or not. So you get an inherent "debounce" delay, which is entirely unnecessary.
Debouncing keys correctly works so: If the switch generates an edge, sent key down/up event immediately and ignore further switch transitions for 5-10 ms. There is no point in waiting for the switch to surely have finished bouncing before reporting the event, because if it is bouncing it MUST have been pressed/released and you know which one it is because you know the prior state of the switch.
---
Compositor delay due to VSync
Obviously compositors are using double-buffered vsync precisely because they intend to limit FPS to $smallValue in order to save power and prevent the 3D hardware from entering a high performance power state. They really should be using triple-buffered vsync, but only start to render if something changed, resulting in much lower latency without constantly running at 1000 FPS. There should be a way for the compositor to be notified of changes in client areas, since stuff like X11 damage protocol and RDP are a thing.
I suspect in practice most software is not very good at giving the compositor the info it needs to avoid wasteful refreshes of the whole desktop, and the compositor probably also isn't putting in as much effort as it could.
I think the best a compositor might be able to do is something along the lines of keeping track of the latest possible time it has to start rendering such that the frame can be swapped to the front and sent to the screen. This should result in the lowest possible average latency while rendering at the vsync rate. DWM is clearly not doing this (since the delay is discretized to 16.7/33 ms), the Linux might be doing this, or it might be rendering on every damage event and use triple buffering, either would be plausibel given the 8 ms average delay.
Thinking back, the first time I noticed input lag was on a Mac LCIII running Word in the mid-90s. Then for a long long time I didn't come across any particularly noticeable latency, except on really crappy websites and in really crappy Java apps. Then Microsoft bought Skype and started working their magic on it, and this seemed to open some kind of floodgate of high-latency crap. That's not even a decade ago.
After that, little by little, everything seemed to slow down noticeably. We've now reached a level when this is seemingly normal. Even programmer colleagues who are my age and older are looking at me like I'm curious when I complain about the latency. I'd say something toxic about Electron here, but it's prevalent in native programs as well.
Have things really gotten so much more complex since 2010 that we can no longer put a character on screen in a timely fashion?
These abstractions, however, are leaky, and almost all of them are leaky in the temporal sense. There's a pretend continuity or constant throughput that's not really there.
Everyone knows that operating systems run short time slices of processes on physical cores, so that each program can "pretend" that it runs continuously on bare metal. But of course, not really, so there are gaps in the flow of execution that can occasionally be perceived by end-users.
If that were the only sin of pretense, then that could be worked around, or carefully tuned, but the reality is that it's just one of many layers.
The garbage collector of managed languages (e.g.: JavaScript in Electron) pauses execution within the process too.
Even unmanaged languages have variable overheads when allocating or de-allocating from the shared heap.
The desktop window manager helps each application pretend that it has a rectangular surface from (0,0) to (w,h), when in reality that is transformed and overlaid. That can introduce a lot of variation, particularly because the DWM has its own threads and its own garbage or heap.
The video card in turn is no longer just a block of memory mapped into the address space of the program doing the drawing, but is its own little computer with cores, threads, schedulers, locks, clocks, delays, and so forth.
The display in turn might further delay things because it has complex overdrive or scaling logic, so it needs to buffer frames.
It's turtles all the way down.
The Amiga 1200 I mentioned earlier does pre-emptive multitasking on a 14 MHz CPU with 256 bytes of cache. I can switch between my IDE, my paint program, the OS desktop, a file manager and a simple text editor without any noticeable delay. It's as fast as flipping between desktops in FVWM on my PC (an operation which, incidentally, never seems to suffer from latency).
> The video card in turn is no longer just a block of memory mapped into the address space of the program doing the drawing, but is its own little computer with cores, threads, schedulers, locks, clocks, delays, and so forth.
The graphics architecture of my Amiga consists of several different chips all timed to a PAL signal and, since they're sharing memory with the CPU and other I/O, are also affected by constant interrupts.
> The desktop window manager
There's a DWM on my Amiga as well, called Intuition, providing several abstractions for programs to open screens and windows and render graphics and text in them. Plus, of course, GadTools, the system library for drawing UI widgets.
> The display in turn might further delay things because it has complex overdrive or scaling logic
The cheap, modern flatscreen connected to one of my Amigas upscales and upsamples _and_ does A->D-conversion on the analog RGB signal and yet manages to show my double-buffered displays scrolling in one pixel increments, with 50 Hz vsync without stuttering or tearing.
Yes, the layers of abstraction have increased in number and complexity, but so has the speed of the surrounding architecture. My PC's clock speed is more than 100 times that of the Amiga, it has 4000 times more RAM (in fact the caches in my cheap CPU exceed the amount of RAM on the Amiga), displays are now connected to the GPU with a wide-bandwith digital interface, and so on.
All of this could perhaps be valid excuses if it was consistent. Yet typing in a Firefox <textarea> feels faster than typing in for example FocusWriter, and typing in an xterm faster still. I can paint smooth freehand curves in Gimp with instant feedback (something the Amiga is not always capable of, depending on how much bandwidth the selected resolution requires). The computer is capable of full screen, full frame, fully vsynced full HD movie playback without stuttering or dropping frames.
The most interesting aspect is of course that a computer that might feel laggy in certain applications is fully capable of emulating an Amiga, complete with the perceived snappiness of the UI, despite all the overhead of emulation _and_ the supposed delays of the surrounding architecture.
(edit: getting some numbers straight)
If you're using a 60Hz monitor, you're only going to see new frames every 17ms at most (1 second / 60Hz). Your graphics pipeline might have some additional buffering, adding 10s of milliseconds of lag. Your monitor likely has some input processing as well, adding anywhere from 10-20ms before it sends the frame to the physical display. Add a few milliseconds here and there for input processing and even the response time of the physical pixels, and you're looking at something like 50-60ms minimum for total display latency before you factor in the software.
Using 144Hz FreeSync or G-Sync monitors can shorten that update time, but you're still looking at 30ms end-to-end latency in even the fastest setups, and that's before you account for software processing lag.
The difference between Sublime Text responding in 11.4ms average and Atom responding in 28.4ms average is difference of almost exactly 1 frame of latency for a typical 60Hz monitor. Add up all of the other sources of lag (buffering, monitor input lag) and you're looking at something like a 5 frame latency instead of a 4 frame latency. Still less than the blink of an eye (literally).
From another perspective: If you really believe that you're sensitive enough to feel a difference between something like Sublime Text's 11ms processing latency vs. Atom's 28ms latency, then you might want to invest in a proper 144Hz gaming monitor with low input lag, as it would improve your experience by the same margins. A gaming-specific keyboard might also help, as average keyboards can have 10-20ms of input lag before the keypress registers with the OS (Source: https://pavelfatin.com/typing-with-pleasure/ ) Realistically, though, I doubt many people could A/B test the difference between a 1ms and a 30ms latency editor under ideal conditions, let alone while typing out some code.
That is looking at the average latency for Atom. If you look at the max latency Atom, it is 60ms compared to 15 ms for Sublime text. That is basically 3 frames of latency difference on a 60Hz monitor, and would definitely be notable for the average person.
Having done some UX work in this area, I can say that people greatly overestimate the effects of latency on text entry when they're just looking at the numbers. End-to-end lag is more obvious in situations like dragging objects, but even then people manage to work around it.
Typing isn't performed on a tight feedback loop in the brain. We don't wait for the letter to appear on screen before starting the process to press the next key.
Typical human reaction types are on the order of 250ms for a simple visual stimulus. Recognizing letters will take even longer. A difference of a few 10s of milliseconds isn't generally going to be noticeable for typing unless you're really going to great lengths to A/B test.
Consider that people can SSH into remote machines all of the time with 100s of milliseconds of latency. The experience may not be optimal, but our typing doesn't fall apart over SSH.
I've experimented with video games with various amounts of display lag or dropped frames; nothing blinded or anything (although, setting up a blind test sounds fun), and there's clearly an increase in difficulty the farther you get between input and response. Writing code is clearly not Mario Brothers, but small delays can add up.
Indeed! I don't have a problem using say a SSH console with 1 second lag. After a bit of initial cursing, it works just fine, I can write my code or edit conf files just fine. It's not as comfortable as writing at home, but it's not really a big deal.
However if the lag is inconsistent, say due to packet drops, it's horrible. Even if the base latency is low.
You get latency-free editing because the file is edited locally and sent to the server on save. Simple implementation but highly effective.
This is more in the 1-2ms range, even for non-gaming devices.
Example: https://www.rtings.com/monitor/tests/inputs/input-lag (Scroll down to the non-gaming monitors)
The gray-to-gray response time for the physical pixels is in the neighborhood of 2-4ms, so the additional latency comes from the processing.
Example of a 9 year old "office" monitor: https://www.prad.de/testberichte/test-monitor-dell-u2412m/6/...
It's pretty obvious when using extended shortcuts or a string of keys that are committed to muscle memory, like Ctrl+Shift+P (+ first letter of a cmd) or Ctrl+K combos.
Imagine playing an instrument wired up to headphones and then adding a 30-60 ms delay... the longer the delay the more it creates that "off" feeling
Don't forget too, those are bare metal calculations. Stick those apps inside a VM like many devs do or add some other latency along the I/O chain and it all helps to create a perception of lag.
We need to keep these numbers in context. 30ms delay would appear as an echo if you were playing an instrument and also listening to a 30ms delayed version of the same sound.
However, you're not feeling the latency. You're noticing the presence of two distinct signals. That's why UX input latency matters much more when you're doing something like dragging across the screen with your finger on the screen: You visually register the distance between your finger and the expected location.
However, when you're typing a memorized series of letters on a keyboard, you're not comparing the keypresses to the letters appearing on the screen. You already know what you're typing and what you expect to see on the screen. Visual stimulus latency is on the order of 250ms, so you wouldn't even begin to start processing types for an order of magnitude larger than the latency of these text editors, for example.
Most of us are operating with regular 60Hz laptops or monitors. End-to-end latency from physical key press to physical color change on your monitor might be as high as 50-60ms even with zero-latency software.
nothing makes me want to throw my computer through the window faster than typing and not having an instant response honestly. It doesn't matter that the brain cannot process the letter. Would you imagine handwriting where the shape of the letters you write is 1/4th of letter shape late ?
Even for typing, just comparing xterm at 60hz and 120hz on my monitor feels different when typing moderately fast (100 wpm according to https://typing-speed-test.aoeu.eu/?lang=en)
Not necessarily; it could be an electronic instrument.
I once tested a "e-piano -> midi -> PC -> headphones" chain. The latency was not really noticeable when I pressed only a single key, but actually playing was impossible. After I replaced a (software) component with something with lower latency, it was bearable. While I cannot precisely estimate the latency reduction, I would guess that it was less than 30ms.
The eyes on the other hand have very little response to this. Some will say they can spot a 15ms vs 30ms visual delay easily, but this is an open debate rather than an obvious fact. A 15ms delay in audio is noticeable to almost anyone.
Different senses, different sensitivities. Comparing them isn't very instructive.
I've made a jumping mechanism for a game that makes you move faster if you jump right when you hit the ground again. This system does not give any audible cues as to when you should jump. When something in this system is slightly off, ie keyboard latency, jump height, gravity, etc. You can't time the jumps properly and you're left feeling that you somehow just can't do it but can't really pinpoint what the problem is exactly.
I think the same applies to many games. Especially fighting games.
With that said, I don't think writing code is about timing. I think it's nice when an editor feels very responsive, but I don't think it necessarily makes me more productive.
All this measures seems to be time from injected keyboard event until pixels change on whatever bitmap/surface in memory. Message passing, in other words.
Using an emulated legacy API (like Windows GDI) might yield a very low latency score in this test, even though it might take much longer to actually be visible on the display.
That's the reason you won't see a Java-based pacemaker anytime soon.
That said, it allows companies to use cheap web developers to build (previously more pricey) desktop apps, so it's used everywhere now. It's good enough to make money, but obviously using the wrong tool for the job will make some aspects of the experience worse.
Since most consumers hate working with the buggy software on their computer anyway, the loss for a company in making it slightly worse by introducing delay is negligible.