The protocol is public. Signal software is free and open-source. Its mobile clients, desktop client, and server are all published under the AGPL-3.0. Any modifications for the app binaries and protocols would be quickly noticed. Hackers and cybersecurity companies are not fools.
Would it though? The Signal server source repo was once not updated for almost a year until somebody noticed and called them out on it:
Signal doesn't support reproducible builds, does it?
I don't understand why Signal is not pursuing the reproducible builds. It looks suspicious. Verification of a binary takes a huge effort and can only be done by knowledgeable people. Case in point: nobody noticed or cared about the lack of undisclosed binary updates of Signal without released sources.