But why should it swap things out even when I'm not crunching data and there's no memory pressure? I'd rather everything stay in memory and stay fast, since I have enough RAM.
You can tweak this, it is indeed a local-system configuration thing that a Linux user should know and think about - and maybe - if your distro has such a thing as a policy about it - this ought to be considered as well.
For example, I use Ubuntu Studio, precisely because it is generally well set up, swap-wise, to absolutely not interfere with audio performance (note: this DAW has been rock solid for decades) .. but every distro has its own 'medium ground' to consider when setting this.
The thesis/antithesis for/against swap is un-useful. Swap is a knob, not a switch.