I do something similar to @diggan, but expand with globbing because I'm probably lazy:

  NODE_PATH=$(dirname ~/.nvm/versions/node/v6*/bin/node)
  export PATH="\
  ${NODE_PATH}:\
  ...

  export NVM_DIR="/Users/jbergstroem/.nvm"
  [ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"

Also, the delay that a lot of you may be experiencing has to do with npm, not nvm.