The problem I find with something like gnome, is that it pulls in a huge amount of dependencies, and it isn't always easy to cleanly remove it.
It's not GNOME's fault though -- it's an annoying issue with big metapackages. Same thing happened with XFCE on a stock Ubuntu install.