Distrust of bankers is deep-seated and centuries old. In times past, usury was a sin.
Obviously there are important advantages to society from having a modern financial sector, but the public is not entirely wrong to be suspicious of the ill-gotten gains of financial wizards who conjure profits by shuffling numbers around rather than performing more obviously useful work.