I quite agree that it's grown a bit gnarled and tangled with optimizations over the years.
It's interesting to compare the current version to the earliest available copy of the annotated source code, from version 1.1.1, seven years ago, in a simpler JavaScript age:
https://cdn.rawgit.com/jashkenas/underscore/1.1.1/docs/under...