What good will an encrypted home folder do if your computer gets compromised? It protects against offline, physical attacks, sure, but I fail to see how it is safe against the myriad of remote attacks there exists.
The only threat a passphrase protects against that an encrypted home doesn't is someone walking up to my unlocked laptop (it locks automatically after 5 minutes, and I make a point of locking it when I'm leaving). If you run ssh-agent, you're exposed to this threat anyways.
EDIT: After some thought, passphrases can protect against a poorly executed attack, where just files are copied wholesale, or an accidental leak. So while theoretically you're not protected, it may be sufficient for some practical security. I'll be looking into using ssh-agent in the near future.
Use a yubikey or smart card to level up from these.