the two are not mutually exclusive. Consider the a the number of former colonial powers is small compared to the number of former colonies.
The only difference between a normal government and a colonial government is the degree of integration. If you live under any government you are ruled, not free. The countries we have now grew by conquering and accumulating territory -- that is except in the cases where territories decided to give up their independence like the states in the US.