For those who have never heard of ControlMasters, I highly recommend them. SSH can multiplex multiple sessions (you invoking ssh at the terminal) over a single TCP socket — this feature is called ControlMaster. The first ssh command takes the normal amount of time, but every command after that is just ~a round-trip. No slow asymmetric key exchange. If you close all your connections, there a (configurable) timeout until the ControlMaster's connection closes.
You can combine this fact with zsh's autocomplete powers, and get pretty-close-to-instant (on a good connection) tab-completion of directories on the remote server, which is extremely nice when trying to scp something, as you can tab complete the paths in that command.
(It also saves a few PIDs on the server, as a single sshd child deals with all your connections.)