Presumably you were raised to think you have an impact on the world; many people don’t feel that way at all.
Many people wouldn't mind living under an oppressor who only imposes their ideologies/religion/way of living on you. e.g. the Mughal empire in the subcontinent might have imposed their religion in many instances but they treated the country like their own home. It is the stunted growth and several famines under the British empire that made many regions more prone to extreme poverty and conflicts.
Often enough rulers are exploitative regardless whether they sit in London and fill their pockets or in the country's capital.
This is why colonial states tended to be far more brutal than normal dictatorships: ultimately, they don't need to convince anybody that their rule was legitimate, because their core support is global, rather than domestic.