Yep, you have to trust the secure enclave for this to work!
Basically all it does is given Signal plausible deniability for public search warrants:
https://signal.org/bigbrother/
It would do nothing to stop national security letters. Basically, you should assume the functionality provided by the secure enclave is only in name; given the NSA ATT national security letters enabled both physical access and system modifications, see no reason to believe others would not be required to do so as well:
https://en.m.wikipedia.org/wiki/Room_641A
Again, average person does not have a threat model that makes this relevant, but to say Signal does not have access to the SGX data is purely based on trust, not mathematical proofs.
If your threat model does not allow it, use manual e2ee with pigeons. Otherwise, well secure enclave will get better with time, which will make Signal better too.
Intel already dropped SGX from their new line up of CPUs (11th gen +).
1. SGX is just an implementation, could be replaced by something else. 2. SGX still works on CPUs that have it.