Thank you for pointing this out!

Having my fingers leave home row as little as possible is vastly superior to reaching for arrow keys and having to reposition my fingers on home row.

With vi that made immediate sense to me. Switching to HHKB layout made even more sense, because I now can reach both CTRL and backspace easily without lifting my fingers from home row.

I did change my keymap even further, so that arrow keys are mapped to fn+hjkl. This way, I can navigate everything just with hjkl (: