I always like to point out that the predecessor to the Western civilization, the ancient Mediterranean, including Greek city states and then later on Rome, was famously built upon slave labor. Even the democratic Athens had a large slave population. However, in a Greek city state like Athens (or the earliest form of Roman legions) the free men -- the slaveowners of the society -- were also the hoplites fighting the wars the city would be involved in. Being a citizen of such a city-state is one the origins of the Western concept of citizenship.
The distinction if you are able to view your community as "us", towards whom you have obligations and duties, not as "them" who rule over you.