I'm with you, and no, it's not paranoia, especially when stories like this (also on the news.yc front page right now) are still making news: http://www.venturecake.com/layered-technologies-hacked/
From the article:
Major US hosting company Layered Technologies have been hacked. Credentials for 5-6000 hosted accounts - and the data stored in them, including customer details in web stores - have been compromised.
I hope Mint doesn't used Layered for their hosting.
With Wesabe you download a desktop app, put you bank credentials into that, and then the desktop app uploads your data to the Wesabe site. With Mint you give their webapp your bank credentials and they store them on their server.
I'm not saying Mint is screwed but current phisihing scams and security lapses have people scared. Financial sites will develop a stigma that many people have when you ask them to 'install' something on their machines.
I actually stand in line to cash cheques instead of using the ATM because I'm afraid it'll mess something up somehow.