Computer Latency at a Human Scale

From my experience a non-continous user action with a response under 100ms is perceived as "instant". Continuous input is different, and humans can perceive individual frames at 16ms for sure.

The video/audio is well studied, I forget the number but it's pretty short, this is out of my butt but I think it's around 20ms?

Keep in mind frame drops are different from average FPS. Every frame taking 16ms will feel far smoother then most frames taking 8ms but with spikes of 80ms (which is far more common).

>I know in game dev, I was always taught that 60fps is fine, but then there are gamers swearing up and down when they drop below 140fps.

My 144 Hz monitor is a literal game changer. I can move my field of view very quickly and the background environment stays crystal clear throughout, making it easier to spot enemies.

I remember reading about the Sega arcade game Daytona USA. Instead of back to front or front to back rendering, it was a mixture of most important (your car, race track) to least important (background stands, trees, etc). It included a hardware watchdog that would trigger the next frame even if the current frame was not finished rendering, to guarantee a constant 60fps always. For an immersive experience, it was more important to keep the frame rate constant than it was to render every polygon.

> I know in game dev, I was always taught that 60fps is fine, but then there are gamers swearing up and down when they drop below 140fps. Would be curious to see what the max allowed latency is here as well.

The 60 fps mark isn't really latency related. Instead, at 60 fps animation without motion blur feels smooth. At 30 fps you need motion blur or it feels like it stutters (and in 24hz cinema the camera produces natural motion blur due to shutter time).

The latency aspect is more complicated, as there's a long feedback loop: The frame is shown on the screen, is perceived by the user's eye, the user's brain computes a reaction, the brain sends a signal to the hands, the hands execute the reaction, the input device perceives this and sends the result to the computer, the next frame is computed and finally it's send to the screen and displayed. Each of these steps takes on the orders of milliseconds (often tens of milliseconds). Faster refresh rates and more fps improve some of them, and in theory any improvement in fps and refresh rate makes the feedback loop faster and thus improves performance. There are diminishing returns because you don't control the entire pipeline (and the parts that happen inside a human are particularly slow), but going from 60 to 140 fps is a big enough improvement that it matters.

Another factor is the stability of the latency: humans can compensate for quite high latencies by predicting the correct inputs. But this only works if the latency is consistent. Just imagine trying to hit anything with bow and arrow if you don't know how fast the arrow will fly. That's why frame drops hurt so much: every dropped frame is a giant latency fluctuation.