i've seen people using them, and if i were of a less honourable persuasion i could abuse that quite easily... on the other hand, its impossible for me to steal information from out of their brain (so far at least).
It's easier than you think to steal passwords from people. Just ask for it!
http://www.veracode.com/blog/2013/03/hacking-the-mind-how-wh...
I use passpack to store different random passwords on all my online services, EXCEPT for my email account. That one is stored only in my brain.
Because now you need to steal two things. My password file and my password.
Now admittedly the one weakness is that once you have both those things you have access to all my passwords, but on balance I believe it is an acceptable compromise.