Working with early-stage PhD students and helping supervise them has convinced me that the job supervisors have is legitimately hard. Sure, the supervisor wouldn't have anything to show for their efforts without students to do the grunt work, but understanding what the frontier of research is, and figuring out how to push that frontier in achievable ways that are relevant to the rest of the community, is extremely difficult. Most supervisors I know work insane hours to keep their labs running.
My issues arise with the personal mistreatment I've received from my supervisor -- none of what makes his job hard requires making abusive comments directly to me or other profs about my personality and work, nor does it require refusing to work with me for >12 months on the paper describing a project so we can get the work published.