CA-signed certificates don't prove you're talking to who you think you are either as any CA trusted by your browser/OS can sign any certificate.
If we encourage users to blindly accept self-signed certificates (giving us end-to-end encryption but sacrificing identification), nothing would stop those actors from altering your HTTPS sessions as easily as they alter your HTTP sessions today. It's throwing the baby out with the bathwater.