RVM can generate wrappers for Passenger. So you use a separate gemset for each app on the server, generate wrappers for each gemset, and then in your web server config you pass the right wrapper to each Passenger directive. You can't use more than one Ruby with Passenger, but you can use as many gemsets as you want.
You can probably do similar things with unicorn, mongrel, etc, but I'm not sure how those work :)