SwiftUI doesn't help, but each panel is a separate process. The previous version of System Preferences hid the delay by showing an animation, the new one doesn't.
Unless it's actually SwiftUI taking time to render that UI, which would be bizarre.
As a macOS user I am so used to everything happening in the blink of an eye, so this is something that stands out. It's really not a big problem, 300-400ms.
Was I imagining that wrong?