So the way I currently have it the man-in-the-middle is useless because once a client connects to the SSH server all other IP addresses are locked out from that port.
I agree about congestion, but IIRC the TCP connection timeout defaults to 75s so since I'm tolerant to connections across two windows (i.e. two minutes) then I think I'm safe.
John.