The key point is that a package is stored entirely under /Programs/packagename - bash is stored under /Programs/Bash/3.0, and /bin/bash is just a symlink to /Programs/Bash/3.0/bin/bash.
When you want to uninstall bash, just use rm -rf /Programs/Bash/3.0.
They also have some tools which allow you to cleanly install programs from source in this manner, i.e. programs that are not included in the Gobolinux distribution. So regardless of whether Gobo packaged it, all programs are under package management (unlike yum/apt-get/etc).
What I am wondering is how they solve the dependency problem, like say something depends on bash and you remove bash with rm do you get any warnings or help or just a broken system? (This is more for when it is part of gobo and not installed from source).
Although I think using the filesystem for this purpose is a good idea -- you get parallel-installed versions for free, and it's more reliable and transparent than what you get from rpm -- you're right that just deleting stuff willy nilly won't work. You still need to manage dependencies, and dependency graphs are not easy for humans to keep in our heads.
I compile stuff from source all the time, I just run rpmbuild -bb foo.spec instead of typing configure, make install manually.