In the specific case of text editing with vi bindings, having up and down on the same row is more convenient, since you don't have to move your fingers to perform linewise vertical motions (compared to a standard WASD "grip"). Navigating within a line using left and right is uncommon when using vi bindings, since there are usually better ways to jump between locations within a line, so it makes more sense to pay the cost of having to move your fingers in this case.

Not only that, but I posit that any context that uses vi-style binds uses vertical motions much more often than horizontal ones (pagers, editors and other applications with vi bind ports, etc.).

As for games, whilst WASD might be more intuitive, it is technically inferior in tactical shooters, where having the W and S keys on the same column means your (admittedly somewhat rarely used) forward-motion counter strafe will always have an inherit delay due to having to move your sole finger dedicated to those two keys. In this case, vi-style ASDF is again marginally better.

That being said, your gripe seems to be overstated, if this is your biggest problem with vi-style binds.