I have a little trick that makes nvm not impact your overall startup time.

First, put the version you commonly use in your path

  export PATH="${PATH}:/home/victor/.nvm/versions/node/v6.5.0/bin"

Then, instead of sourching nvm when your shell starts, do it when you call an alias instead.

  export NVM_DIR="/home/victor/.nvm"
  alias n=". $NVM_DIR/nvm.sh" 

Now, you have a node+npm version by default, and you can do a quick flow if you want to change

  $ n
  $ nvm install 6.6.0
  Now using node v6.5.0

'n' has always worked perfectly for me. Takes all of twenty seconds to get up to date, and switching between versions is stupid simple.

I do something similar to @diggan, but expand with globbing because I'm probably lazy:

  NODE_PATH=$(dirname ~/.nvm/versions/node/v6*/bin/node)
  export PATH="\
  ${NODE_PATH}:\
  ...

  export NVM_DIR="/Users/jbergstroem/.nvm"
  [ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"

Also, the delay that a lot of you may be experiencing has to do with npm, not nvm.