This sounds like swap needing to be swapped in and then released. Check your memory usage.