I actually suspect that's a fallacy. When you learn & use something new, it's actually confusing when it's similar to something else. I have no issue with typing either Qwerty or Dvorak, but Azerty, where only a few keys are different from Qwerty, drives me nuts.
But if you have to switch between two layouts regularly then you definitely want something completely-distinct rather than something close-but-not-quite.
It seems like colemak is in the awkward position where you wouldn't even consider it unless you're already comfortable with qwerty, but you're also in a position to stop using qwerty entirely and switch all your keyboard layouts.
"Easy to learn – Allows easy transition from QWERTY. Only 2 keys move between hands. Many common shortcuts (including Ctrl+Z/X/C/V) remain the same. Typing lessons available."