You're doing too much. All you need to do is:

  source /usr/local/rvm/environments/$BUILD

You can put it in a wrapper script if you like. It will unset all conflicting environment variables first.