That's why the Web of Trust, or classic GNUPG key signing parties are a forgotten/ignored must have. Anyone can change and go rouge of course, but it's statistically less likely.
Of course, it's far from being 100% effective, but it mitigates the issue significantly.