Yes. ssh-agent prevents non-root users, including the user running ssh-agent[1] from being able to extract the private key. That's much more secure than storing unencrypted keys in ~/.ssh.
[1] ssh-agent uses some tricks, like making itself setgid, and/or using prctl(PR_SET_DUMPABLE, 0) on Linux to make its memory undumpable.