I'd dig into what modes (minor and major) are setup on the files you're editing.
I will also say, there was some way I'd once configured org-mode that really slowed it down when there were a lot of code blocks (as it could switch into the appropriate major mode for that block). I don't recall ever "fixing" it, it just got faster over time. I was specifically using org for literate programming. When using it straight with primarily the outliner, dates, todo lists, etc. it didn't have the same issue.
The one thing I've noticed that really helps with load times is running emacs as a daemon on startup and launching new instances with ```emacsclient -nc```. But that doesn't address issues while it runs.
I use emacs exclusively for org-mode. I think I don't see the problems you see.