Exactly. You can have a rock-solid password login system, but pretty much everyone has a reset feature which kicks a link to the email address, and allows that user to change the password. It's no less secure than that.
A password-less login, the email can be deleted and unless there are logs of last login, and the user notes the pertinent detail, then the intrusion can be covert.
Having someone unknowingly have a key to your apartment is much more of a breech than discovering a broken lock that you know needs fixing.