I once spent a while adding timings between each item in my zsh config files to find what was slowest.
The nvm init was the worst of it, so now I have an alias for `$NVM_DIR/nvm.sh ] && \. "$NVM_DIR/nvm.sh" && \. $NVM_DIR/bash_completion` I just run manually before using nvm, rather than letting it slow me down half a second or whatever it was every time I open a new terminal.
It's worth poking around if things get slow enough to be a pain. There's a decent chance it's 1-2 things that you don't even care about all that much.