> Have I been told to leave somewhere [...]
Why do some black people hate white people? Because their great grandparent perished on a cotton plantation, because their friend was hunted by some white folks in a truck, because their friend was choked to death by a white policeman, because your race is targeted by disenfranchisement.
Why do some white people hate black people? Because they hate black people.
There is a substantial difference between those two, and the difference is central to the definition of racism vs. racially-charged. Someone being a vengeful asshole does not make them racist, someone being an asshole to a race in general does not (necessarily) make them racist. Racism is hating a race, it is not being angry with a race that participated in horrific acts against you and your (in some cases near) ancestors.
Is it fair that you are being held accountable for the acts of your ancestors, and other living white people? No, that isn't the claim. You find the retribution you face unfair/unkind/unjust, as you have the right to. Now imagine facing the same for merely existing. The claim is that you couldn't possibly understand the experiences of a black person. The claim is that you do kinda' have to accept the stories of black people at face value.
The claim is that the vast majority of their life is defined by racist experiences, where you have a handful of racially-charged experiences to point towards.