Historically people used to sell compilers - so minimizing installation to dev machines probably was a savings (and in those times space was at a premium everywhere).
That said - I am with you, give me any other programming language besides shell!
But people have told me stories about working for the government
(my background was more "big tech", and video games, which are both extremely different)
Some government/defense systems are extremely locked down, and they don't have C compilers
So people make do with crazy shell script hacks. This is obviously suboptimal, but it is not that surprising in retrospect!
FWIW, there is an apple.com .dmg "command line tools" that is much smaller than Xcode formal, but I actually came here to say that to the very best of my knowledge a $(ruby -e $(curl ...)) to brew install will then subsequently download pre-built binaries from GitHub's docker registry