I never felt pained on switching layouts. If this makes any sense: learning the muscle memory for Colemak almost seems to be on a lower "level" than vim (to me), so I kept hitting 'i' and 'a' and so on without even noticing.
I think that it's better to let the muscle memory of Colemak to develop and you will eventually get used to vim.
When I use vim I'm constantly thinking about words, like 'ci' (change in) 'ya' (yank around) etc.
So when you get your Colemak right vim is not a big hassle.
I'm curious about h, j, k, and l. Same story?