If you're doing that to move vertically, you might want to try vim-sneak[1] instead.

Personally, I disable the line-numbers column altogether (and display the current column:line in the status bar).

[1] https://github.com/justinmk/vim-sneak/