He doesn't actually spend all his time at the factory. He's mostly in LA. There are people on Twitter who track the flights of his private jet. He's mostly in LA, he'll go to the bay area maybe for a day a week. More rarely to Reno. If he's spending all his time working, it's at SpaceX of anything. But more likely he spends his time at home searching his own name on Twitter.

The insane work hours stories are just part of the mythology.