I'd say that disk comes a close second, if not first. Overtaxed slow disks with low memory makes for a terrible desktop experience.
Once your memory usage climbs and your OS starts paging/swapping things to disk, seemingly trivial operations that page in new code will take forever.