What scares the hell out of me, it would be so easy to backdoor an implementation of the open core in a well-hidden way.

Whereas the "big" CPU providers are staking their reputation and therefore future business on providing a non-backdoored CPU, it would be fairly trivial for an individual device manufacturer to provide a backdoored CPU design for their chip design.

It could become the whole cheap-device OEM firmware situation all over again (as we saw with many backdoored routers), but this time the blob is located on-die, so is significantly harder to reverse engineer or audit.