While other posters are right to point out that server certificate should make sure nobody can truly hijack your sshd there's no point in taking the risk unless for some reason all privileged ports are in use.
Not to mention that ssh's key model is a bit broken since there's not built-in way to distribute the keys/check the keys against an authority like with SSL. Most people just accept the key they're given when they first connect (I know I do).