I feel like not counting the military is silly in that case, because the military in colonies was essentially involved in police actions, engineering duties, and ensuring administrative rules were followed.

As an essential part of the colony, you can't really count them out. That's similar to how in feudal structures, and military regimes, I'd count the military and civil service as one entity.