Intel's implementation is considerably worse than that. Even if you assume the hardware itself isn't compromised, every remote attestation has to go through the "Intel Attestation Service" which has no end-to-end protection. The IAS is what actually validates the enclave's signature, and it returns a "success" or "failure" message which is signed with an Intel key. But there's absolutely no technical measure that prevents Intel from being compelled to sign a falsified response; a client would have no way of telling the difference.
This is documented by Intel [1] and I'm hardly the first to notice it [2] but people still seem to talk about SGX as if compromising it is equivalent to backdooring the CPU, which is inaccurate.
[1]: https://software.intel.com/en-us/articles/intel-software-gua...
[2]: https://www.blackhat.com/docs/us-17/thursday/us-17-Swami-SGX...