I did that exact thing in a project last weekend, and ended up spending 5 hours trying to fix the horrible mess that ensued. I only tried doing this in the first place because bundler errored out on installing one of the gems, and I figured a "clean" local setup would fix things.
For some reason the project still wanted to use global gems, and when I tried to delete them to force it to use local gems I broke everything. In the end I just installed RVM with a clean version of Ruby and let it use global gems.
I'm sure I did it wrong, but the errors I got were just insane. I couldn't delete gem X because I had gem Y installed, I couldn't delete gem Y because I DIDN'T have gem Z installed. I couldn't install gem Z to delete Y to delete X because my installed version of gem Q didn't match the version bundler wanted, but I couldn't install a new version of Q because of reasons...
After reinstalling the exact same gems (through RVM instead of the apt-get version of Ruby I had before) things still didn't work, and it took me quite a long time to figure out that the error "your version of [some YAML gem] is wrong" means that you have to add a line of Apache config telling Passenger to use Ruby.