> groups who frankly almost nobody wants to have associations with
I'd caution you about this claim; Gab, Spinster, Glindr, some of the less savory Japanese instances probably fit what you're saying. If you want to be less charitable about most people's broadmindedness, toss in the furries. But from what I've seen, a large chunk of Mastodon users tend to just be LGBTQ. For this group, the control/independence is a lot about knowing what you can expect your moderators to take seriously (both in terms of allowed content and protection against harassment). I'd question the term "fringe" for this group. There's also been a movement away from Twitter onto Mastodon of Indian users upset by Twitter's acquiescence with Indian government demands. In that case, it's not so much about privacy as making legal demands a game of whack-a-mole.
Otherwise I totally agree with what you're saying.