Not the person you're replying to, but I agree that for me (as another heavy shift-G user), vim-racer having `:relativenumber` turned on is where my own troubles are coming from.
I'd really love to be able to specify whether I want absolute or relative numbering (or both or neither); the :set commands don't seem to be implemented?