Switching does have its benefits but I'm afraid it'll be a pain to deal with it switching the vim keys around.
I think that it's better to let the muscle memory of Colemak to develop and you will eventually get used to vim.
When I use vim I'm constantly thinking about words, like 'ci' (change in) 'ya' (yank around) etc.
So when you get your Colemak right vim is not a big hassle.
I'm curious about h, j, k, and l. Same story?
Shai Coleman had a really extensive Vim remapping setup, but he has since stopped using Vim: http://colemak.com/pub/vim/colemak.vim
inoremap lh <Esc>
Before I used k-j, but the position of those in Colemak is 'e-n'... so it's totally unusable.
I mean if you're typing 'entropy' it leaves insert mode, and you get mad :)
Didn't take me more than a week to get used to 'e-n'.