Yes I've hit this as well! Such an annoying behaviour.

However I think it's a good habit to make records in `~/.ssh/config` for each of your servers anyway just to keep tabs what, where, who, and with what keys.