I always got the impression these ultra rich are workaholics that barely see their family.
Elon is the CEO of like four or five companies I think, while also ostensibly heading a government agency. If you can have four or five full time jobs, then they are not full time jobs.
Given that, I suspect that they're able to find plenty of time for themselves.