To make this concrete, consider expertise on England. Suppose Prof. Smith has never been to England, but has studied it in detail for decades. And suppose Mr. Chav has grown up in England, but is uneducated and isn't even sure who is the Prime Minister. Who is "allowed" to talk about England? Smith, who knows all the facts, or Chav who lives there and knows what it's like? Can Smith tell Chav that he's wrong about England, or does Chav's lived experience matter? (I'm using England as a less controversial example, but this usually happens in discussions of race or gender.)