Did you just forget that CAs exist? They are centralized. You always have to trust SOMEONE. Even if it's the person that wrote the CA software being used, or the supply chain that provided the software to a vendor, or or or. See what I mean?
Yes you do have to trust someone and the CA is the trusted entity for doing the verification, but once they do the verification and in effect encode that verification onto a certificate, their role is done.