I'm not a Colemak user, but from my experiences with Dvorak, I have a theory on why this happens. Precisely because it shares many keys with QWERTY!

Explanation: when I type in Dvorak or QWERTY, the only hesitation is when I type the first letter of a text I need to type. Then, a muscle memory kicks in and I type without thinking. The only exception: words starting with letter "A". This letter is special, because it is the only key that has the same position in both Dvorak and QWERTY!

So when I type words starting with "A", my muscle memory kicks in not on first typed letter, but on second! With Colemak, I guess this effect might be in full swing for almost every letter you type, and THAT would be a significant problem. So what is sold as being Colemak's advantage is probably a disadvantage if you need to type on different types of keyboards.