having worked with ruby & rails both for about 7 years, i've run into my fair share of gem hairballs.
in general, i've had _more_ problems on osx than on linux, mostly due to various ARCH flags required by c extensions.
that being said, the "system rubies" on every linux distro i've used are generally infinitely horrible, and this is likely the case for the poster. system packaged gems are also insanely gross as well.
one of ruby's saving graces is that at least it's not used by the base system for initialization, which has bit me more than once installing a python that conflicted with the system python (though that was many years ago, things may have improved).
rvm has its own issues (and seems to be getting more complex with every rev), but in general, makes for clean living. bundler is nice too.