It makes a big difference. The FAA talk about "flow patterns" for correct activities in certain phases of flight; training to recover from some non-normal situations is quite literally learning a specific muscle memory, as well as decision making (albeit quite simple decision making).
Something that is quite hard to convey is how you actually _do_ get into an "OODA loop" in an aircraft (https://en.wikipedia.org/wiki/OODA_loop) – observe, "orient", decide and act. For example, if flying VFR a lot of the time is spent looking out the window and actively checking that there is "nothing there". You look at the instruments, but you don't stare at them. A large part of flight training is pattern recognition, muscle memory, and fine motor skills.
I spent a lot of time gliding while writing up my doctoral thesis (mostly as it's hard to panic about not working when you're too busy not dying) and found it very helpful for other situations in everyday life. Part of that is knowing, exactly, without looking, where everything is. For that reason, aircraft have physically differently shaped handles for the important knobs so that they are unlikely to be confused. On the glider I have most hours in, the lever on the left that is blue, with a moulded grip is the speed brake. The tow / winch release lever is a yellow ball. In many prop aircraft, the throttle, mixture and prop pitch knobs are again given different shapes (round, star and "undulating" respectively). Knowing where to move your hands to is actually a really important skill and the keen-beans that X-plane sells to likely have set-ups where not only are these things in roughly the right place, but the distance between them and the chair is measured. Why? Well, if you're learning to fly an aircraft that burns ~$150 per hour (or more), and you can practice a ton of circuits in X-plane, it is a hell of a lot cheaper to try to get good at it outside of the aircraft first.