It takes no appreciable time to adjust when switching between any of the three. Maybe for a split second I notice the mouse cursor and then... nothing.
If I were gaming on the XDR it'd be one thing, but it's a productivity monitor. I don't need 144hz (or more) to watch text scroll, and the improved pixel density is 100x more satisfying for productivity usage.
In fact if anything I'd say going from 6k to 4k is what's more jarring than going from high refresh to low...
Similarly, I see a lot of upset over mouse acceleration curves but I adjust to differences within 5-10 seconds of switching at most and forget about it.
There's some out right now but they're either crappy gamer panels, very large and low density, or extremely expensive.
The other big gotcha this monitor has is that like every monitor Apple has made for the past 20 years it only has a single input. It's really tough to find another monitor these days that doesn't have multiple and just can't see any real upside to not having at least an HDMI port (for your Apple TV of course) available.