This may be an issue with ssh trying to resolve reverse DNS on your server - you can check by turning off UseDNS in ssh settings and logging in again.