Signal, and the Signal server operators, malicious or not, do not know the phone numbers of people you are communicating with (presuming the secure enclave on the server isn't cracked). Signal does know your phone number, so someone could figure out if you use Signal. If you're worried about that, or personal sharing of your number, you could falsify information to create an anonymous phone number, or you can just use Matrix or Wire.
At least with GPG I can factory reset all of my computers and phones and not have to re-establish trust if I take the right steps to preserve the secret key information.
Even if I don't preserve that correctly, people change computers less often than phones.
On the other hand, I'll admit that my GPG key is newer than my Signal number (which I've owned for 15 years), due to upgrading crypto algorithms.
I perform the safety number verifications in exchange for forward and backward secrecy. GPG isn't enough to establish a truly confidential communication channel, I think. (Unless you erase keys after every sent message, maybe.)
I'm not happy with Signal's dependence on a phone. Ideally I'd like to use a pocket-sized SBC for secure messaging. Come to think of it, that sounds rather like a phone. Just, uh, without the cellular hardware, and with a user-installed OS.