It's like if you go into a bar and there's a guy screaming a bunch of curse words and racist crap in the corner at everyone near him. Normal people are going to leave if no one kicks the guy out.
The real reason X is being attacked is because the Following tab doesn't inject propaganda like every other social media / news website does, other than ads of course.
I still use X/Twitter weekly and I don't have any problem with the posts I see. My problem is with comments, which are usually where I will see hate and spam.
There are two distinct twitter feeds: 'for you' feed and 'following' feed. The former's like a sewer, but there's no excuse for the 'following' feed being anything but pristine (since it's curated by the user).
Something Musk discussed when he bought twitter was how users didn't always know the difference between the two feeds since the labelling was not clear - there was a 'stars' (?) icon to switch between the two.