A bunch of people deciding not to be your friend isn't censorship.
When many instances co-ordinate to block an instance based on activism (Fediblock), they are censoring everyone on the targeted instance, many of whom may be innocent.
People deciding "whether they want to follow someone" is completely different from what's happening here.
It's the equivalent of sending an e-mail to my neighborhood listserv saying "the guy who lives down the street is a bit of a dick, let's not invite him to the chili cook-off next time."
And censorship power is further concentrated by the existence of Fediblock