For instance the colonial office (the bit in London that administered the colonies centrally) had 14 staff in 1816, and grew to about 25 by 1850

Important context:

1. The Colonial Service never administered India, the biggest colony.

2. Lot of these colonies had representative institutions of their own. "We have 43 colonies, and not 50, 60, or 70, as has been variously stated; 27 of these have representative institutions, some in a higher degree of perfection than others, no doubt, but the whole have representative institutions, including Canada, New Brunswick, Jamaica, Antigua, Barbadoes, and all the minor West Indian colonies." [0]

[0]: https://hansard.parliament.uk/commons/1849-04-16/debates/807...