How can you ever tell that by port alone? If a box has been exploited, it's been exploited.
Seems like something a little script or patch could fix up really easily - make sure your daemon is running on port 1234. If it's not, take whatever defensive measures you think would be effective.
If you're worried about an impostor sshd on that host then I would tend to agree that it's simply not prudent to be connecting to that server at all, let alone passing key material or credentials.