The absolute best way to know that is to know the public host key and verify your keys. Additionally, you don't have to resort to high ports to achieve a best of breed. Just !=22 is enough to avoid majority of skiddies.