Even assuming a secure implementation of this system (unlikely), what's to stop a man in the middle attack? When you're browsing at the coffee shop, I just lie to you about the server's public key.

Now sure, you could access the server one from a trusted connection to get the public key. But then when someone spoofs the public key you get a large and nasty error message saying the public key has changed. And history tells us that security can't safely be reduced to that type of user interface problem.

Long story short, you're massively underestimating how valuable it is that web browsers ship with certificates installed. When in doubt, don't re-implement SSL. ;)