If latency is your biggest concern you have to use a terminal emulator without double buffering. So far xterm and mlterm are popular choices.