Your distro may vary on the policies around this, but the way the stack generally works, is that whatever shared libraries, and their memory resources, which haven't been touched in a while, are put into swap. So if you've run a few programs, left them open or whatever, switched the UI to other things and whatever over a period of a day, or even a few hours/minutes (its a knob), then bits of those programs will have gone cold. The local libraries and static things that haven't been executed - in a while - will move into that qualification.

You can tweak this, it is indeed a local-system configuration thing that a Linux user should know and think about - and maybe - if your distro has such a thing as a policy about it - this ought to be considered as well.

For example, I use Ubuntu Studio, precisely because it is generally well set up, swap-wise, to absolutely not interfere with audio performance (note: this DAW has been rock solid for decades) .. but every distro has its own 'medium ground' to consider when setting this.

The thesis/antithesis for/against swap is un-useful. Swap is a knob, not a switch.