Unfortunately, the HSM vendors are generally just as shifty as a typical SaaS vendor, so don’t believe their promises of secret inviolability if you can get uninterrupted time alone with a device.
Realistically, any decent shop manages all of these risks pretty well, but there’s a loting rigmarole to do it robustly.
In some ways attacking the homes might be easier than attacking Apple Park. (Just thinking out loud...)
Private homes of the ultra wealthy and powerful are usually protected in non-obvious ways that most houses on most streets are not, to put it mildly. It turns out that machine guns are legal for private citizens (such as a 24/7 armed private security team), even in California, if you are wealthy and connected enough.
But assuming that's not the case now, my point still stands about how it would be maybe easier than attacking Apple Park, relatively speaking. Though, according to Apple's WWDC videos, invading Apple Park to swap an M1 between a MacBook and an iPad is easy when you are the CEO and wearing a rubber mask ;)
And if Xbox was doing that 20 years ago, I'd expect Apple to be as well.
It'd make sense for Sony, but some of their approaches to software security have been lol inducing, so it wouldn't shock me if they aren't.