A passwordless key gives anyone with acces to that file, access to the login associated with it. If that file is inadvertently exposed (oops, checked it into github...), any machine you have a login on must be considered compromised.
For example if you login remotely to a machine, and want to access a git repository on another:
$ eval `ssh-agent -s`
$ ssh-add ~/.ssh/id_<yourkey>
$ ssh -A <firstserver>
you@firstserver$ git clone git+ssh://<secondserver>/path/to/repositoryAgent forwarding should be enabled with caution. Users with the ability to bypass file permissions on the remote host (for the agent's UNIX-domain socket) can access the local agent through the forwarded connection. An attacker cannot obtain key material from the agent, however they can perform operations on the keys that enable them to authenticate using the identities loaded into the agent.
Consider using a dedicated key for each of those circumstances, set sane defaults in your ~/.ssh/config on all machines, and be very careful about what ends up in any of your ~/.ssh/known_hosts files, as they provide a road map to other destinations.