Some of the allegations involve minors, people whom are lured into economically dependent conditions, and group pressure situations where consent is not so easily defined (no matter how much motivated reasoning is abused to rationalize breaking with 'outmoded conventions' like not seducing subordinates).
But part of it is also a more general point: Their plan is to build an "AI God" constructed according to their own moral theory before anyone else, and that it will prevent anyone else from creating a competing god-- the absurdity of this plan doesn't prevent the effort from creating terrible harm (e.g. the corporate monopolization of AI, mass surveillance on unimaginable scale, etc). The AI safety faction sees itself as moral guardians of humanity's future-- and it's absolutely relevant that many people would consider them weird and that some would consider them (rightly or wrongly) outright morally repugnant.
You lose your right to have no one else worry about what you do in your private life when you start making plans to take over the world, even if you think it's in everyone's best interest. Every other modern authoritarian dictator also though their actions were in the public's interest.