At least windows doesn't freeze your whole desktop under heavy IO.
Edit: I'm getting downvoted by people that never used Linux probably.
Edit: I'm getting downvoted by people that never used Linux probably.
1. dm-crypt threads are unfair to the rest of the system's processes [1]. On dmcrypt systems, regular user processes can effectively raise their process scheduling priority in a multithreaded fashion by generating heavy IO on dmcrypt storage.
2. Under memory pressure, the VM system in linux will enter a thrashing state even when there is no swap configured at all. I don't have a reference on hand, but it's been discussed on lkml multiple times without solution. I suspect the recent PSI changes are intended as a step towards a solution though. What happens is clean, file-backed pages for things like shared libraries and executable programs become a thrashing set resembling anonymous memory swapping under memory pressure. As various processes get scheduled, they access pages which were recently discarded from the page cache as evictable due to their clean file-backed status when other processes ran under pressure, and now must be read back in from the backing store. This ping-ponging continues dragging everything down until either an OOM occurs or pressure is otherwise relieved. This often manifests as a pausing/freezing desktop with the disk activity light blazing, and it's only made worse by the aforementioned dmcrypt problem if these files reside on such volumes.
For some reason they seem to be prioritizing the supposedly improved dmcrypt performance over fairness under load, even though it makes our modern machines behave like computers from the 90s; unable to play MP3s and access the disk without audio underruns.
I assume it's because they're not hearing enough complaints.
Things may have changed over the years - I’ve been running a Linux desktop recently and haven’t seen this kind of issue yet (the kind where you need to use magic keys to ask the kernel to kill all, sync and reboot) but reading your post, perhaps this is because RAM is much more plentiful these days.
It does even much weirder things. https://blogs.technet.microsoft.com/markrussinovich/2007/08/...
At least I'm going to give them credit for disclosing what the problem was. I would've be too ashamed to admit it.
But honestly, that could also be the NVMe drives.
With bfq-mq, I've encountered no desktop freezing under heavy I/O.
It does it as well on Windows and unlike Linux, Windows still has a risk of permanent damage under low space available, it was even worse in the XP days but these issues are still there.
In turns out that the cause of this problem, known as bufferbloat, is the accumulated effect of excessive buffering in the network stack, mostly the system packet queue, but also includes the routers, switches, drivers and hardware, since RAM is cheap nowadays. The TCP congestion control works like this: if packet loss is detected, then sends at a lower rate. But when there are large buffers on the path for "improving performance", the packets are never lost when if the path is congested, instead, they would be put into a huge buffer, so TCP will never slow down properly as designed, and during the slow-start, it believes it's on the way going to the moon. On the other hand, all the buffers are FIFO, it means when your new packets have a chance to get out, it's probably no longer relevant, since it takes seconds for moving it from the tail of the queue to the head, the connection would be timed out already.
Solutions include killing buffers and limiting their length (byte queue limit, TCP small queue), another innovation is new queue management algorithms: we don't have to use a mindless FIFO queue, we can make them smarter. As a result, CoDel and fq_codel are invented to implement "DELay-COntrolled queues", they are designed to prioritize new packets that are just arrived, and dropping old packets to keep your traffic flowing.
And people realized the Linux I/O freeze is a variant of bufferbloat, and the very same ideas of the CoDel algorithm can be applied to the Linux I/O freeze problem.
Another interesting aspect was, that the problem is NOT OBSERVABLE if the network is fast enough, or the traffic is low, because the buffering does not occur, so it will never be caught in many benchmarks. On the other hand, when you start uploading a large file over a slow network, or start copying a large file to a USB thumb drive on Linux...
https://lwn.net/Articles/682582/
and
There are myriad causes for poor interactivity on Linux systems under heavy disk IO. I've already described the two I personally observe the most often in another post here [1], and they have nothing at all in common with bufferbloat.
Linux doesn't need to do less buffering. It needs to be less willing to evict recently used buffers even under pressure, more willing to let processes OOM, and a bridging of the CPU and IO scheduling domains so arbitrary processes can't hog CPU resources via plain IO on what are effectively CPU-backed IO layers like dmcrypt.
But it gets complicated very quickly, there are reasons why this isn't fixed already.
One obvious problem is the asynchronous, transparent nature of the page cache. Behind the scenes pages are faulted in and out on demand as needed, this generates potentially large amounts of IO. If you need to charge the cost of this IO to processes for informing scheduling decisions, which process pays the bill? The process you're trying to fault in or the process that was responsible for the pressure behind the eviction you're undoing? How does this kind of complexity relate to bufferbloat?
I've had similar experiences. On Windows, where either through bugs or poor coding, an application requests way too much memory, leading to an unresponsive system while the kernel is busy paging away.
On Linux I've experienced the system killing system processes when under memory pressure, leading to crashes or an unusable system.
I don't understand why the OS would allow a program to allocate more than available physical memory, at least without asking the user, given the severe consequences.
You can tune the overcommit ratio on Linux, but if memory serves (no pun intended) the last time I played with eliminating overcommit, a bunch of programs that liked to allocate big virtual address spaces ceased functioning.
If I hit 80%, I get 10-20 second lock ups.
If I hit >95%, I get 1-2 minute lock ups.
Using Ubuntu 18.04.
On windows things may be unresponsive, but at least ctrl-alt-del responds, and at least the mouse moves!
The main difficulty is I can't tell if my machine has crashed vs is overloaded if the UI doesn't do anything for several minutes.
Well, why can't distributions that come with recommended GUIs just set that high-than-normal priority by default?
I don't quite understand why the system can't keep what it needs to perform REISUB in memory at all times. Surely that's a tiny program!?
Feel free to file bugs for your preferred distro. It would be especially appropriate to do this for critical UI processes like xorg/wayland or lightdm, and for "lightweight" desktops like xfce/lxde that aren't going to cause resource pressure under foreseeable conditions, even when run at higher-than-normal priority.
Are you sure you've set up the magic sysctls correctly? Ubuntu ships with magic oomkill disabled, by default.
On all machines I've tried, whenever I've needed it, magic oomkill has always worked, and I've been thankful of the fact that it's implemented down in the kernel.
I attribute this to the fact that the darwin kernel has a keen awareness of what threads directly affect the user interface and which do not (even including the handling of XPC calls across process boundaries... if your work drives the UI, you get scheduling/RAM priority). I don't think the linux kernel has nearly this level of awareness.
You're talking about priority inheritance in the kernel. In Linux, this is in development as part of the PREEMPT_RT ("real-time") patches, already available experimentally in a number of distributions.
Case in point: I recently tried unzipping the Boost library on an up-to-date Windows 10, and after trying to move the frozen progress window after a minute, the whole desktop promptly crashed. I have to say, the experience is better than it used to be, because at least the taskbar reappeared on its own. (Decompression succeeded on the second attempt after leaving the whole computer well alone ... but it certainly took its time even on a high-end desktop computer.)