1. The security implications. Without a passphrase, anyone who manages to snare a copy of your private key can use it to access any servers where you've authorized it with no further credentials needed. Not good. Using a passphrase turns SSH auth into a poor man's two factor system: what you have (the key file), and what you know (the passphrase).
2. It breaks agent-forwarding, which is the system that gives you the ability to SSH onwards from the remote machine to other machines without (again) being asked for your password. This has practical implications when using SSH to drive things like application deployment over git, when you have to ssh onward from the server to your git repository. Doing things this way sidesteps the awkward use of deploy keys, and is actually easier to set up and use as well as being more secure. With agent forwarding, there's no need to keep your private key anywhere except your 'home' machine, and yet you get the same benefits your would if you had it everywhere.
The annoyance of having to type in your passphrase every time is greatly lessened by the use of a key agent on your 'home' machine. OS X has provided a key agent since 10.5 I believe, and stores the passphrase in your Keychain (you know you're doing things right if an OS X dialog box prompts you for your passphrase when running SSH in the terminal). Gnome has one built in as well, and ssh-agent is always around for people who want to roll their own.
If there's interest, I'll write up a blog post about how to get this stuff set up. It's not really easily discoverable stuff, but it _is_ really simple and useful once you get it up and running.