The binaries are open source, but Google doesn't design their build chain to recompile from scratch every time.

Also, you don't need to compile all of AOSP just to get the toolchain binaries.