As someone else who learned and still uses Colemak... it's just not worth it in the long-term unless you're using it to address something specific like chronic RSI.

QWERTY does return to you, sort of, but only if you still actively use it in some other contexts. If you don't use it regularly, you'll lose proficiency in it and not really be able to easily switch between the two (though in my own experience I did retain some of the muscle memory, it was just much more error-prone). I'm at the point now where I can somewhat easily switch between the two, but it's very context-specific. My fingers recognize my laptop keyboard and gaming machine keyboard and instinctively use Colemak and QWERTY for them, respectively. But there are still times where my brain gets into an "undefined state" where I can't figure out which layout a machine is using or which layout my fingers are attempting to type, and as far as I can tell I end up typing something like a nonsensical hybrid mashup between them which is surprisingly sticky and difficult to recover from.

In the end, I believe my typing speed is something like 10%–15% faster than what it used to be with QWERTY, but I don't feel there's many contexts where I'm genuinely limited by typing speed. On the other hand until I become semi-fluent in QWERTY again, it was maddeningly difficult to use other people's computers.