This is exactly the key core distinction. The purpose of the state is to be the most powerful organization in the room - to constrain other actors. It’s imperative, therefore, that it be democratic and representative. Notably, part of the instinct to break up other large organizations is to prevent them from assembling enough resources to have a supersized impact on the state - the problem with monopoly is that monopolies buy out their competition and neuter regulations, the problem with wealth disparity is the ultra wealthy are sufficiently powerful to move the state in the direction they want it to go.
I agree with you generally regarding reducing the overall size of governing bodies and I agree with Terrence about the benefits of small organizations and the drawbacks of large specifically around the investment and perceived ownership of members of those organizations, but having a small state fundamentally requires having small organizations everywhere - and anti-monopoly, antitrust, and anti-wealth concentration - because for the state to be democratic and representative, it must be the most powerful organization in the area it covers, otherwise it’s just a tool for the more powerful to use.