Downside existed before Let's Encrypt, it just got amplified with it.

General public does not differentiate between the SSL certificate validation level.

Let's Encrypt provides domain validation certificates, which only validates that one owns the domain in question.

There is another level - Organization Validation SSL certificates, which involves manual checking that this is the legal entity it claims to be. I would expect the financial institutions to use this kind of certificates to avoid phishing, but sadly I've seen some of them use Let's Encrypt.