Makes sense to me, I felt the exact same way.
I think that it's better to let the muscle memory of Colemak to develop and you will eventually get used to vim.
When I use vim I'm constantly thinking about words, like 'ci' (change in) 'ya' (yank around) etc.
So when you get your Colemak right vim is not a big hassle.