Since then I've been able to switch between the layouts swiftly. At work I use Dvorak on my external keyboard, but when I take the laptop with me I switch to QWERTY. The switch is a mental flip akin to talking in another language (I'm fluent in Russian and English).
I think the switch is very worth it. If I was re-doing it, I'd probably learn Colemak instead of Dvorak (apparently it's a slightly better choice for some reasons).
Nowadays I'm only fluent in QWERTY and Carpalx since I stopped using the other two once I made my decision but switching is no different for me than when it was 4.
Anyways to answer your question now that the background is out of the way: Switching for me seems to be more a function of how comfortable I am with the layout than anything else by far and I've become extremely comfortable with the layouts I use. I can switch between QWERTY and Carpalx instantly and I very often do e.g. when a game was coded based on the letter sent instead of the scancode or when I'm in an RDP session to a prod box.
The only time things can get jumbled for me is if I'm working in RDP with QWERTY on one monitor and another layout locally. Jumping between I often forget it's a different layout because I haven't consciously switched by triggering the input method toggle keyboard shortcut like I normally would. The solution is simple though: If I'm going to spend a lot of time hopping between the RDP session I'll just toggle to QWERTY globally until I'm done.
I'm a swiper on phones so it makes sense to leave it qwerty since that's what the swiping algorithms target.
I had no trouble switching between layouts, and could type 60wpm in each, in consecutive minutes. I tested this when I had my computer in the family room connected to the family computer to transfer files (parallel cable!). I had one keyboard on the desk and the other on my lap and had no difficulty with the different layouts while working on both computers simultaneously.
I never got much faster in Dvorak than QWERTY, but it was much better for my hands on the keyboards of the time.
But I think it's possible to maintain 2 layouts in your muscle memory