If not, can an attacker gain access to your email server or DNS with your ssh key? If either is true, they now have access to everything not protected by 2FA that uses an email address they now control.
There's so many things to get right.
You can design a system such that there's a very high likelyhood even a nation-state attacker won't be able to intrude without leaving traces - if you make no mistakes.
Or you know, you could also just rotate your ssh keys in addition to everything else. "I have logs" is really no excuse to forego something that is this easy.