Actually, privileges are exceptions to the rule, and they each have their own justifications. The rule as stated is fine, and any privilege must be specifically justified on its own.
Attorney-client privilege and psychiatric privilege exist for different reasons. Attorney privilege comes from the necessary functioning of the judicial system, as you pointed out. Psychiatric privilege, in the opinion of the U.S. Supreme Court, "serves the public interest, since the mental health of the Nation's citizenry, no less than its physical health, is a public good of transcendent importance"
What is the reason that a banking privilege should exist? What "public good" is being served by protecting private transactions that outweighs the harm done by evasion of tax laws or other regulations? That's a difficult question to answer, and simply saying "it's the same as other privileges" is not sufficient.