Every subsystem goes "ah buffering for a few ms or a frame doesn't hurt anybody" and in the end you have click-to-photon latencies of 150+ ms.

This starts with incorrect debouncing in input devices (which can cause 10+ ms delay on its own even in dedicated "gaming" hardware!), slow/inefficent poll rates, wrong input handling in games (many still seem to receive input on the "main" thread ticking along at the frame rate, which causes huge amounts of input lag at lower framerates and also clamps inputs to frametimes, which causes significant position errors), incorrect V-Sync implementation (like the broken "triple buffering" in D3D9-11 games, this should be a thing of the past nowadays), graphics drivers favoring throughput over latency as they are almost exclusively benchmarked on average fps (used to be that they could buffer up to 9 frames, or ~150 ms) etc.

Input lag is the period between input and output, not just the time to poll the input. E.g. on 250 Hz standard mice just polling the mouse will average 2 ms input lag with 2 ms jitter and that's before you've even done anything with the input. If you don't have a high refresh rate VRR gaming display and don't want tearing the same repeats, e.g. 60hz would be 8.3 +- 8.3. That's halfway to 20 and we haven't even gotten to delays from the game code where you can choose things like triple buffering for higher FPS at the cost of another frame of latency. Input lag can also include output to the monitor and delay from the monitor depending how it's being measured.

In this case it's more about the rendering pipeline input lag so the USB polling delay and the monitor output delay aren't counted, the swapchain delay to prevent tearing or trading latency/FPS are though and then you have to add in your actual game still.

Those are End to End System latency. Your USB Input Devices and Monitor Refresh Rate, excluding anything in between, could easily had 20ms delay in the worst case scenario. A 60Hz monitor would have had 16.6ms of worst case latency alone.

There are a variety of factors that add up to many ms of lag. USB and HDMI are a few that come to mind. USB is incredibly complex when compared to e.g. ps/2. If you are building an embedded board like rasberry pi and you want to add ps/2, you just add a couple pins. If you want to add usb, you buy an extra chip.