No, you have to trust anyone with access to the server; secure enclave provides no security given it appears it is vulnerable to side channel attacks; see comment you just replied to for related link.
Basically all it does is given Signal plausible deniability for public search warrants:
https://signal.org/bigbrother/
It would do nothing to stop national security letters. Basically, you should assume the functionality provided by the secure enclave is only in name; given the NSA ATT national security letters enabled both physical access and system modifications, see no reason to believe others would not be required to do so as well:
https://en.m.wikipedia.org/wiki/Room_641A
Again, average person does not have a threat model that makes this relevant, but to say Signal does not have access to the SGX data is purely based on trust, not mathematical proofs.