that's a setting you can turn off. settings -> desktops and spaces -> reorder spaces
The setting is "Automatically rearrange Spaces based on most recent use" which explains why the behaviour felt so intermittent.
However, swiping beetween the previews, it sometimes jumps to random places in the order - which is not nice.
Possibly a bug, but I might as well just write this as a letter to Santa because it's got more chance of being read than a feeback.