The problem is a "real" certificate being MitM'd and replaced with a self signed one. Things like certificate pinning might be able to mitigate that. Even though it's been a bit of a farce (in no small part due to the quizzically large set of trusted root CAs) the idea is that the CAs that you trust do their due diligence and don't sign fraudulent requests.
>[0] If you really care, you should use a CA-signed cert. But every attack that possible when using self-signed certs is not only possible when using plain HTTP, but much easier to execute, and also much easier to execute silently.
Right, but the owner of the server you're connecting to is the one that has to choose if the data they're serving or receiving qualifies for protection of this sort. By omission they've basically taken the position that their pages or data are not worth preventing an attacker from MitM'ing you. If you allow self-signed certificates to fly without a warning at that point you're taking the decision to secure from a MitM attack away from the server owner that wants to protect their pages. Sure, a savvy user would notice a change, but what if they've never been to that site before?
Here's a scenario: Bob's Widgets sells Widget 9000 and has a secure site with a certificate signed by a CA. Alice hops on free WiFi at a Widget convention and visits for the first time, hoping to buy some of these amazing widgets Bob sells. Except Mallory intercepted the traffic and swapped out a self signed certificate. Bob's only way of telling Alice that his certificate and therefore traffic is authentic is by using one signed by a trusted third party.
Now that isn't to say that I don't think your idea has merit. It just doesn't fit into how we currently do things. What might work is a keeping port 80 and port 443. Port 80 is secured by a self-signed certificate but the user isn't informed (via their browser's indicators, like a green icon or what-have-you) that their connection is secure because it isn't in the sense that it will protect them from MitM attacks. It is however encrypted and you can have the browsers trigger warnings when the key changes similar to the way SSH does. 443 still behaves exactly the same where users only trust CA signed certs by default.