You have 10 bits of entropy at best, unless you put it above 1024, at which point if it dies, any none privileged user on the box can sniff passwords.
Do you actually have untrusted users on the box?
Why would you not secure the custom port to root-only?