If you don't have a way to confirm that the key you're seeing from the other site is right, you're inherently vulnerable to a man-in-the-middle attack which removes the benefits of the encryption against the attacker.

https://en.wikipedia.org/wiki/Man-in-the-middle_attack

httpS://en.wikipedia.org/wiki/Zooko's_triangle

It's not clear that the certificate authority system was or is the best solution to this problem, but it is a problem that calls for some solution. In the case of Domain Validation, we only try to confirm that the key is appropriate to use with the domain name, which is the smallest possible kind of confirmation that can be done to address the crypto problem. There's no attempt to validate or verify anything else about the site.