And, even worse, xorg and xdmcp to get py25-pil.

Yes, in most cases, but most packages have minimal dependencies due to Homebrew's "use what OS X provides" philosophy, i.e., the XCode toolchain (Homebrew doesn't even have a package for GCC).

The thing with MacPorts is, basically, you're installing a parallel universe that doesn't know about the software your system already had. So it has to bootstrap completely from scratch, including its own toolchain, basic libraries, etc.

Once you get all that, it goes faster, but the first time you go to install something simple you're basically going to walk away for a couple hours while it sets up its own little world.