A passphrase defeats that threat. And integrating the ssh agent with something like the gnome keyring means you never even have to remember your passphrase.
My initial reaction is this: Couldn't malware on my laptop also monitor my keystrokes when I unlock the key? Or when I log in to my VPS web interface? I mean, if the goal is to have a malware infested computer that is no threat to external systems, it seems like there are tons of other files/apps/system you'd also want to password protect, to the point of making the computer almost impossible to use.
Still, it's an interesting point. SSH keys are more sensitive. I do keep an extra password on my password manager.
once malicious third party code has executed on your machine as you it's game over.