This can become problematic if you're using an SSH agent and have 4+ keys loaded into it. The problem is that your agent will step through keys to try auth with, but most OpenSSH servers will reject keys after 3 failed attempts. This means that you also need to keep an up-to-date entry in ~/.ssh/config as to which key to use with which system.