Does "But if you're going out of your way to look suspicious" advance the conversation?
But banks aren't mandated to admit you either. Just because it's legal, doesn't mean a private establishment has to let you in. When it comes to denying entry, banks are relatively tame. Some establishments go beyond that, by denying entry unless you wear formal clothing, or presenting proof of identity.