Exploiting a single user's account and opening a high port is not the same as getting root on a system.
Seems like something a little script or patch could fix up really easily - make sure your daemon is running on port 1234. If it's not, take whatever defensive measures you think would be effective.
If you're worried about an impostor sshd on that host then I would tend to agree that it's simply not prudent to be connecting to that server at all, let alone passing key material or credentials.