The problem is that the definitions used to censor speech often experience 'scope creep' and start to encompass more and more, up to the limits of whatever the circumstances and pressures of the time allow. Usually that means the limitations are set by whomever is able to gather more political power at that moment. And by 'political power' I mean not just explicit political power from public positions, but also power encompassed in actions like mass action on Twitter. These dynamics are usually influenced by a small vocal minority and not agreed to by everyone, but the resulting actions still impact everyone. Hence it is not a great way to manage the exchange of ideas.
PS: you don't know me, so I am not sure what standing you have to talk about my privilege or why it is relevant.