The safety number is essentially equivalent to a self-signed certificate. You can use it to consistently identify a key holder, but it doesn’t offer you any way to identify who that key holder is. If you want to trust a self-signed cert, then figuring up whether you should or not is entirely up to you. This is the problem that CAs address with CA signed X.509 certificates.
If you want to validate identity as a service, then the only options you really have available are a central authority, or a web of trust. Both of which have serious downsides, and neither of which are offered by Signal.