Power is always expressed through violence. Someone wants to use the land you're currently using. They leave it alone, even if their situation is miserable. Why? You can call in the cops who will subject them to violence and make their lives worse. What if they can take the cops? Well, wherever you happen to be, it used to be part of someone else's domain; it was taken away by violence. Violence shapes society, it shapes history, the threat of violence keeps the world ticking along smoothly, and responding properly to danger helps to ensure that smooth action. your sympathies don't matter.

You're probably right to not be afraid, but things can kick off when the circumstances are right for it. And we haven't arrived at this historical moment by being nice.