Homebrew now has prepackaged binaries ("bottles") for most of the packages that take awhile to build, which has really helped the speed thing.
At least, not using /usr/local myself, I see a lot of packages building because they have detected I changed the base directory.