It is almost as bad to have your passphrase-protected key permanently stored in ssh-agent, because anyone with access to your machine can use the key without the passphrase. A better solution is to use ssh-agent with the -t option to establish a lifetime (after which you will need to re-enter the passphrase).
My setup is to keep ssh-agent running with a 2-hour lifespan, and connect to that automatically when I log in. Basically this:
$ ssh-agent -t 7200 > ~/.ssh-agent
$ echo "source ~/.ssh-agent" >> ~/.bash_profile
$ [... log in ...]
$ ssh-add
I am not annoyed by re-entering my passphrase every two hours.