The surprising part for me is, OS does not recover. As if once it tanks, it becomes more cautious about which memory is paged and which is not, even if there's plenty of free physical memory later.
To let you skip the anecdote if you don't care, I'd be very curious if someone can answer _why_ this happens.
Earlier this week I was running into Very Strange errors coming out of some deep libraries in one of my hobby services. Nothing was specifically failing but things were going very slow and vomiting a bit. I observed that I had over 25g of active Ram (Typically idle at ~8g) and some absurd amount of paged data too. Even after killing almost all non-system running processes I was sitting at ~20g total, including a 10g nonpaged pool and 6g process private. On reboot, and for the week since? 8g. (On Win10)
However, the MacOS kernel seems to require a much higher amount of wired memory than the Linux kernel for similar workloads, so...
I don't know if those are related, but it'd be interesting to figure out what MacOS does differently (their kernel is open sourced) and to see if that behavior could be adapted into the Linux kernel.
Really? I’ve had the opposite experience. MacOS running out of memory and paging heavily almost always necessitates a reboot to get performance fully restored, even if you kill the memory hog (if you aren’t willing to deal with a sluggish machine for a while that is).
I usually just swapoff and swapon, but I'm sure there's a better way.
The real recovery from this is cleaning virtual mem, and let your program run for a while so that some cache is collected. This may take seconds to hours, depending on your application and vmem size.