Are you on Windows? I've heard Emacs is rather bad on Windows in terms of performance.
The one thing I've noticed that really helps with load times is running emacs as a daemon on startup and launching new instances with ```emacsclient -nc```. But that doesn't address issues while it runs.