I can still type just as fast as I could before I switched on qwerty keyboards. I left my laptop keyboard qwerty and just switched my external keyboard to Colemak so I swap somewhat frequently depending on if I'm at my desk or on the couch/out of the house.

I switched from QWERTY to Colemak about two years ago. When I had first gotten fully adapted to Colemak, and my typing speed in Colemak got up close to where my QWERTY speed had formerly been, my QWERTY speed took a big hit (down from 80 WPM to around 20 WPM). But now, two years on, I can touch-type QWERTY fast-enough (around 60 WPM). I kind of have to establish for myself a QWERTY context by starting to type while looking at the keyboard.

Colemak changes relatively few keys from qwerty, especially compared to Dvorak. I use colemak full time and can still touch type qwerty reasonably fast.