Unless I trust each CA, their processes and every employee who could circumvent them, the current CA infrastructure is inherently unsafe. Self-signed certificates are only marginally less trustworthy (rather than having to compromise a CA, a bad actor would simply have to generate a new certificate and hope that I don't check the fingerprint - and I wouldn't check it).