There are other costs than silicon real estate. Those modes and microcode need to be validated, tested and add bloat to the boot software stack.
Just allowing booting directly in 64-bit mode and making 64-bit SMM mode would be enough to avoid boot software stack problems. Riddance of "legacy mode" doesn't require all it.