SpaceX is the only aerospace job I've held so I can't speak from personal experience working at any competing companies but from what I hear you are right in that SpaceX tends to have a pretty negative reputation within the industry as far as work/pay are concerned. Interestingly, this tends to work in favor of SpaceX employees who leave the company. I've heard that while engineers from competitors often times see working here as undesirable, hiring managers are quite fond of ex-SpaceX employees, at least partially due to it being a difficult work environment. I've heard that at the very least, competency notwithstanding, having SpaceX on your resume shows you can handle long hours and stress.
I would also add that in general I think many people tend to think the grass is always greener on the other side. I’ve known both engineers and technicians who have moved on to “greener” pastures at more established aerospace/defense companies only to later tell me that they weren’t all that green afterall. I have a good friend who joined a large aero/defense conglomerate out of state to be closer to family and while the the pay pump was a nice feature at first, its luster wore off after a few weeks when he ran into some bureaucratic issues that he probably wouldn’t have had to worry about here. I also work closely with a fairly senior engineer who has a decade plus experience at JPL and Rocketdyne and flatly states that working here has rekindled his passion largely due to the relatively flat structure. He was absolutely worn out by all the middle managers he had to deal with previously. I think, and this is especially true at the more senior engineering levels, not necessarily for us lowly techs, having the ability to walk over to Elon or Tom Mueller and voice your concerns directly is pretty empowering for most and makes working here very unique in an industry that tends to put massive layers between engineers and management.
I’ll add a counter point so that I don’t come off sounding like a complete SpaceX shill and say that I also know people who have left and are very happy with their decision and their new workplaces, although many, in my experience most, still look back at their time here fondly.
I would also note that the space launch business inherently requires long hours due to its complexity and unpredictability. If you work in space launch, and are responsible for any flight critical operations, you will work the occasional 60, 70, or even 80 hour week, there is simply no way around it and it even happens at places like ULA and Rocketdyne.
This isn’t to excuse SpaceX’s sometimes poor planning and management which end up requiring more of these long weeks than should arguably be necessary, but sometimes it’s just a fact of life in this industry. Hopefully as the company matures hours and schedules continue to stabilize, it’s always frustrating to lose good people because they have to make the choice between spending time with family or work.