And how did you get that public key?
An attacker could pretty easily obtain a valid Let's Encrypt certificate using a BGP hijack.
Also, the CA system is in bad shape - CAs have been hacked and certificates were leaked. Not to mention that some of the CAs your browser trusts are not entirely trustworthy or are located in untrustworthy countries. Oh, and from time to time there are attacks against TLS itself (e.g. https://drownattack.com/)