Certificate transparency logs are likely the only realistic way, but you could make the same argument against your DNS provider. Trust has to start somewhere.

Whether or not something like this makes sense to you is probably a question of your personal threat model.