It's probably from creating a backup copy. Compare performance when invoking 'view' instead of 'vim'.
It could be from your plugins if they're doing something goofy.
I use vim out of the box with no plugins and on windows. Not sure if that makes a difference. I’m also typically not editing the file but searching logs.
You need to turn off syntax highlighting before opening a really large file in vim. Otherwise the syntax parsing takes an age.