Please. I don't know about Spencer's unnamed "followers", but he was not quite what you would call a fan of the colonialism of his day:
> That a government cannot undertake to administer the affairs of a colony, and to support for it a judicial staff, a constabulary, a garrison, and so forth, without trespassing against the parent society, scarcely needs pointing out.
> Moreover, colonial government, properly so called, cannot be carried on without transgressing the rights of the colonists. For if, as generally happens, the colonists are dictated to by authorities sent out from the mother country, then the law of equal freedom is broken in their persons, as much as by any other kind of autocratic rule.
From Social Statics, Chapter 27, Section 1 (available at http://oll.libertyfund.org/titles/spencer-social-statics-185... ).