I've never used hex fiend (and have only needed that type of tool a handful of times), but from my understanding the reason this can be so much faster than a text editor is seeking for new lines. If you need to jump 50% through the file, with a hex editor you look at the size of the file and jump to the middle. For a text file you have to count all of the newlines (seek the entire file). Even if the text editor does clever things, many times plugins (like syntax highlighting) do things in naive ways.

Vim is /fairly/ fast even with very large files.