My understanding is that the slowness comes from program code being removed from memory to free up space, forcing the system to constantly re-read it from disk. I still get the slowness without swap.