When I tried Matrix I found a lot of super right wing tech people using it. The main reason I could ascertain most of them were using Matrix was because they had been banned by other chat platforms for their opinions
This will always be an "issue" of decentralized services, just look at the Lemmy devs over on Lemmygrad, super far left. Anyone can host a server running a protocol, lets not throw the baby out with the bath water just because people are saying something objectionable in one server. If you don't like these people join/make an instance that doesn't interact with them.
I’ve found many nice and welcoming communities on matrix.
The only time I've encountered anything of this description is when I was looking into why a server had been defederated, and then found out it was mostly neo nazis and weirdos obsessed with anime children. I had to actively seek them out to discover this.