However, this only prevents an attacker copying the key to another machine. If the attacker has access to that key, then they have access to your machine. Not much stops them from installing malware and using your machine as a proxy for the session they want to establish to the endpoint (with the additional benefit of appearing to connect to that endpoint from your machine in any connection logs).
It should be noted that ssh, gpg-agent, and OpenPGP smart cards already provide the capability to store an SSH key on a removable smart card. Storing your key on a smart card provides the additional barriers that such a proxying setup would only work 1) while you have the card inserted, and 2) if you authorize that access by entering the PIN (optionally through an un-observable channel if you have a Class II or III reader). Of course such hypothetical malware would still be able to inject commands into a session once you log on to the endpoint, but it would be much harder to avoid your detection.
Of course, most modern machines come with a TPM (right? correct me if I'm wrong), while very few people have an OpenPGP smart card.