I don't even use Xcode at all. I'd much prefer a `brew install` or even if they had their own *.pkg file or something that was just the toolchain.
There's no technical reason why this isn't possible -- the Apple gcc toolchain is, by necessity, available as Open Source.
I use Xcode rather a lot, so this isn't a big deal for me, but it seems like a sane answer for "the rest". Sure, you'd still need Homebrew, but that plus a toolchain is a much smaller burden.
It could be done as part of the homebrew install process. This would be a step forward for everyone. Most OS X devs use homebrew anyway, and for those that don't use xcode this would simplify things a lot.
I hate downloading that massive DMG. I can't even use wget easily because I have to be authenticated. Who cares about what apple decides to do or not do with the tools the community developed for itself? That's what the licenses are for, people. So we can look after ourselves.