For certain things in signal you need to trust the server hardware and suspicious technologies like SGX. SGX is a prime target for a NSA backdoor. That some upper-layer software there is open source (with lots of caveats, e.g. you cannot verify which server version is really running and signal has a history of lagging behind on publishing their "open source" version of the server code) is a distraction at best.