I can't encourage people enough to actually set up their zsh manually instead of running a configuration framework.
I spent a few hours on this about a year back and went from over 2s to about 0.15s for a full run of an interactive login shell (`time zsh -l -i -c 'echo "test"'`). No functionality lost, at least none that I actually used and cared about.
Definitely one of the best quality of life improvements I've made to my setup in recent years.