I'm living happily with this:
"relative number switches - show numbers in insert mode, relative in normal
autocmd InsertEnter * :set number
autocmd InsertLeave * :set relativenumber
"relative number switches - show numbers in insert mode, relative in normal
autocmd InsertEnter * :set number
autocmd InsertLeave * :set relativenumber
set relativenumber
autocmd InsertEnter,WinLeave * :set norelativenumber
autocmd InsertLeave,WinEnter * :set relativenumber
(Also, :set number does not reset to non-relative numbers anymore for me)