I didn't find the switch to Colemak nearly as difficult as OP. The way I learned it was with some linux-based typing tutor at the time, where it introduces one letter/key at a time, with emphasis on touch typing (which is obviously important because most people aren't getting Colemak keycaps). I went cold turkey, spent 10-20 hours on a weekend, and could work well enough in Colemak the following Monday (probably 40-50 WPM). Then maybe a few weeks to 110 WPM, where my time waster during that period was doing typing tests.

Throughout, I never fully abandoned QWERTY. It was certainly difficult to switch between them, but my office environment was collaborative enough that not being able to maintain some level of competency in QWERTY really hurt.

Eventually, I switched back to QWERTY because there were just enough things annoying about "being different"--NOT because Colemak in itself was bad. Things like configuring keybindings for things like VIM, where half the shortcuts are positional (e.g. homerow navigation) but the other half are mnemonics (next) and now in Colemak they can't be both. And just in general, sharing your hardware with others and vice versa. Colemak was beautiful, but sometimes the imperfect standard is better than the perfect thing that nobody uses.

Switching back to QWERTY I was back to full speed in a few days. That, to me, is the biggest reason I'd encourage anyone to give Colemak a shot. You don't just lose your QWERTY muscle memory, it just hibernates a bit and is groggy when you wake it up.