I have the animations off entirely, be warned though, it requires disabling SIP which is a non starter for many corporate environments.

If you install yabai[0] with its scripting addition, you can configure hotkeys to change spaces with no animations. It's pretty powerful, and I have specific keybinds[1] to do things such as transition to a space with a specific application running.

[0]: https://github.com/koekeishiya/yabai

[1]: https://github.com/worm-emoji/dotfiles/blob/140c9fd614ebfc54...