On my iPhone I find everything up to maybe 0.6 seconds to be just fine. Part of this may be being used to using SSH over VERY poor network connections.

I think the biggest frustration with latency is not the latency itself, but rather when latency is combined with the possibility that what you entered didn't register at all and the time pressure of having only 15 seconds of signal to submit your command - situations that often occur when using Internet based sessions in the wild.

The other source of annoyance is trying to "not crash the system". Often when a system presents high latency like this, I tend to consciously pay attention to how many "buffered" inputs I'm sending for fear of crashing something, and that self-throttling can get tiring.