This article breaks it down in wonderful detail: http://renderingpipeline.com/2013/09/measuring-input-latency...
The TL;DR is that on a "modern" stack (USB, display buffering, etc) with a 60hz display you're looking at over 100ms of input latency. You can roughly halve that with a 120hz display.
But it won't ever touch the sub-16ms latency that's possible with a "retro" 8/16-bit console hooked up to a CRT display.
A lot of those games were garbage, but damn... it felt like your brain was wired directly into the machine. A very very cool part of the experience that is being lost to time.