If you're not concerned about losing other animations, you can enable the Reduced Motion setting in Accessibility preferences. It completely removes that animation.
Rather than "reduced" motion, I'd really look forward to "no" motion.
Like I had with i3wm... practically _instantaneous_ switch between spaces/virtual desktops.