I think it's much more likely that something bad will happen during a build with "sudo brew install some-package" than someone successfully hijacking, say, my Safari session and then installing a trojan in my user-writable /usr/local directory. This is Mac OS X we're talking about: if you've written an exploit like that, why would you bother attacking apps in a directory that doesn't even exist on the vast majority of Mac installs when you could just as easily modify something in /Applications instead, and exploit practically all of them?

Anyway, back to the dangers of using sudo with Homebrew or Macports. When you build software by hand, without a package manager, do you ever do it as root? I sure don't. But that's exactly what Macports does in its default configuration, and that's what Homebrew would require if you used it with a non-writeable /usr/local. It doesn't have to be a malicious attack, either: just a simple bug in the build system would suffice. For example, if a "clean" rule in a Makefile has a step that looks something like, god forbid, "rm -rf $(BUILD_ROOT)/" and you're running the make as root, you'd better hope that BUILD_ROOT is not defined to be "".

Ideally, all of these package managers would separate the step of building from the step of installing so that you would only need root privileges for the latter, a la Debian's sbuild and friends. Then using a traditional privileged /usr/local directory would be a no-brainer, for the theoretically smaller attack surface it would provide. But none of the Mac packaging systems work like that, to my knowledge. They go for the convenience of a single build+install command. Given that, I'll stick with my user-writable directory and risk the hijacking scenario.