Rather than fall back on broad principles like free speech, you concoct evermore Byzantine rules about who can say what, given their race, gender, position of power or wealth. It’s unsustainable. It’s the Terror, but using cancellation rather than guillotine.
This approach doesn’t scale. In order to do as you say, everyone apparently needs to have a constantly updated internal graph, categorising people across an ever increasing number of categories and defining an ever number of allowable or disallowable viewpoints. It’s way too complex. It starts to sound like some kind of a psychosis.
A healthy principle, like Free Speech, espoused in a few words wins by being universally understandable. It just comes with a flip side that you will hear things that offend you. I think this is an OK price to pay.