Nasty Lockup Issue Still Being Investigated for Linux 3.18
phoronix.com
phoronix.com
Anything you can do to speed up the occurrence of such a bug should be reached for first because just speeding up the frequency with which the bug appears guarantees that you'll be able to fix it sooner, and to be more certain that you have fixed the bug once you think that you have identified the problem.
You can waste months on little nasty ones like this. I love it though, digging in on a bug and not letting go until the sucker is nailed for good. Especially hardware related bugs (interrupt handlers that are not 100% transparent for instance) and subtle timing bugs can be super hard to fix. But that feeling when you finally find it is absolutely priceless, it's like solving a very complex puzzle and finding the solution. I wished it did not take such a toll on my nights though :)
I don't think the size of the buffer is what distinguishes the good from great, because complexity is combinatorial in nature. It's the ability to mitigate complexity by designing systems composed of simple parts.
I probably live in a dream world, but I would say software bugs of the 'causes deadlock in a OS that is used in millions of devices' type should (maybe even must) be avoided by reasoning about the code. If you find that you cannot reason about the whole system at the time, you either simplify it (python's GIL is a nice extreme example; you never read about race conditions inside Python's core) or you improve your logic.
In my world, you should only need the debugger for hardware related bugs, and then, it often is a combination of intuition and sheer perseverence that solves the issue. Especially if timing is involved, looking at the system can make the problem go away. But even there, having thought about the system will help a lot. For example, your reasoning might have said "between two X-es, we always get at least one Y, so…". You can turn that into a hypothesis that can be tested easily (in theory. Practice can be quite different)
On the other hand, if you must use modern PC hardware, you get a lot of unavoidable complexity (multiple interrupt levels, drivers that you cannot all check do compliance with your model of how drivers work, etc), so you may end up with the case where reasoning about the code isn't really possible.
Beyond that the kernel developers value both correctness and performance. Most of these bugs involve multiprocessing and a huge amount of effort has gone into getting rid of them global locks, but even they aren't easy to reason about at that scale - I think you're missing the common thread in "deadlock" and "Global Interpreter Lock". Pretending that a world could exist where we can simply reason about something like the Linux kernel and then using it to state that devs can simplify or improve the logic is taking a completely unsupported cheap shot and the disclaimer about the real world doesn't change that. I'm not even sure what your motivation for the equivalent of "if you pretend that friction & drag don't exist it's really easy to figure out aerodynamics, so plane designers need to do better" (hint, it's not) is, but it certainly isn't to inform, contribute, or facilitate the debate.
If you don't make rules as to how the parts are to behave, how are writers of device drivers going to know how to make a working driver? Tweak it until it doesn't hang the system in a few days?
And how are you going to fix that deadlock you find during debugging? You may know (simple exampe) that thread T1 holds lock L1 while trying to get L2, while thread T2 holds L2 while trying to get L1, but without guidance on how things are supposed to work, how do you even know whether to fix T1 or T2?
IMO, the more complex the system, the greater the need for simpe rules that limit what different parts can do.
Chances are there will still be lockup bugs, but that is not a reason not to try and avoid them.
The other day I was "backporting" some input patches to an older kernel version. The patches all applied cleanly. The resulting source code compiled. The feature I wanted worked. But it still scares me too much to check it in.
What about LTP [1] and Autotest [2]?
«Linux Test Project is a joint project started by SGI, OSDL and Bull developed and maintained by IBM, Cisco, Fujitsu, SUSE, Red Hat, Oracle and others. The project goal is to deliver tests to the open source community that validate the reliability, robustness, and stability of Linux.
The LTP testsuite contains a collection of tools for testing the Linux kernel and related features.»
Locking and RCU have built-in "torture" tests (CONFIG_LOCK_TORTURE_TEST and CONFIG_RCU_TORTURE_TEST). Take also a look at the tools/testing/ directory in the kernel source tree.
Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 3
CPU: 3 PID: 17176 Comm: trinity-c95 Not tainted 3.17.0+ #87
0000000000000000 00000000f3a61725 ffff880244606bf0 ffffffff9583e9fa
ffffffff95c67918 ffff880244606c78 ffffffff9583bcc0 0000000000000010
ffff880244606c88 ffff880244606c20 00000000f3a61725 0000000000000000
Call Trace:
<NMI> [<ffffffff9583e9fa>] dump_stack+0x4e/0x7a
...
I've had to trace bugs under emulation that took substantial amounts of time to trigger. Thankfully in my case, I was able to serialize the entire system into snapshots. And so by just taking a snapshot every few minutes, the one right before a crash allowed me to be able to quickly reproduce the problem.
Does this crash event occur inside of VMs as well, and if so, could VM snapshots be used to accelerate the bug hunt here?
I know that there are certainly bugs that this kind of technique would never work on. I've hit a few bugs that could only be triggered on live hardware before. But I'm curious if they've tried this kind of approach for this bug yet or not.
You basically have to capture everything possible: all memory, the state of all the CPU registers, the state of all hardware registers, etc. Obviously disk would be a real challenge, where you'd have to keep a delta list of disk changes since program start, or simply not serialize that state. If you miss anything, you can have problems loading states correctly. However, there is quite a bit of tolerance between theory and practice, so if there is something that you really can't capture the state of for some reason (like a hardware write-only register when you weren't logging what was previously written to it), a lot of times you can get away with it anyway.
Because the system I am using is so old, snapshots are only 300KB each. Sometimes I dump them to disk, sometimes I just keep them in RAM. I know that a PC would be much more challenging, given how much more hardware is at play, and because VMs aren't quite the same as pure software emulation like qemu (though you could potentially use qemu for this too), but VM software does implement this snapshot system, so clearly it's possible.
You know when the bug is triggered through the visual output. And what's cool is that by saving periodic snapshots automatically to a ring buffer, you can code a special keypress to "rewind" the program. So it crashes, then you go back a bit, and save a disk snapshot there, and wait to see if the bug repeats. If it does, then you turn on trace logging and dump all of the CPU instructions between your save point and the crash. Then you go to the crash point, and slowly work your way back to try and find out where things went wrong.
Not only is it great for triggering crashes, but if you then broke out into a debugger, you could toy with toggling register and memory contents and then single-step through the instructions to see what happens. It's really substantially easier to debug software under emulation because of features like these.
Processes under DragonFly may be "checkpointed" or suspended to disk at any time. They may later be resumed on the originating system, or another system by "thawing" them.
Linux has a history of out-of-memory lockups where disk caching has tied up most of memory. Linux uses free memory as disk cache. When there's a need for memory, clean cache pages are reused. Sometimes the I/O systems have the cache locked at the moment there's a need for memory, resulting in an out of memory lockup. This has been fixed in the past, and broken in the past.
This doesn't look like front page news until it is confirmed to be a widespread problem.
It would seem prudent to invest a bit more time to understand why a particular thing doesn't work, for such a critical bit of code (I'm not implying I know that this is the cause or related to these new found bugs, but the article mentions it, so I'm just a little surprised).
/* On Xen the line below does not always work. Needs investigating! */
Could be a "tearing out your hair" desperate sort of thing where you cast around for any possible cause....
It would indeed. That's no reason not to post the patch though. Welcome to the imperfect world of software development.
I use ZFS where I can, but for root filesystems on linux machines reiser still seems like the best tradeoff.
I can't say I'd trust my data on it these days.
I heard they fixed it in 2012. Actually there have been a lot of developments in XFS since 2011, I'm not really interested in filesystems but you should have a look at it again if you're interested.