There are plenty of ways to improve security but maintain openness.

I think a good idea might be to have TOFU and self-signed only as a fallback. If there was no initial mismatch, and then upate cert periodically.