Only a tiny fraction of the Federation's population is part of Star Fleet. There are only a few thousand star ships scattered over thousands of light years. Crew complements are rather small, so at most a single-digit million people are on Star Fleet ships. That's comparable to the size of just the current US military personnel but the Federation has hundreds of planets and many billions of people.