That's a good point; British "agnostic" "liberals" (incl. notably Mill), often defended the British task of "civilizing the barbarians" the world over.