The most likely attacks on Signal involve trusted insiders or configuration errors, and SGX mostly prevents these, since to exploit it, you'd need to bribe insiders in both Signal and Intel, or find configuration errors in both of their software stacks.
Collusion is certainly still possible, but it's much harder to pull off, since it typically requires nation-state-level resources to exploit. Signal does actually have nation-state adversaries, but the vast majority of other software projects don't.
(I personally think that remote attestation is the single biggest risk to the free software movement, but I begrudgingly accept that Signal is a very good use case for it.)