Hunting deep-seeded kernel bugs
anchor.com.au
anchor.com.au
strongly disagree! no kernel component should introduce memory corruption, and especially not in response to an ioctl from any user mode code. they say:
> Specifically, various offsets provided from userspace are used by the driver without any checks. If these offsets are maliciously chosen, the driver can be induced to write to arbitrary kernel memory
sure hope they patched that too!
The patch appears to discard the DMA flags coming from user space, and instead applies the ones that are correct given the kernel's knowledge of the buffer.
But yes, this is ultimately a kernel bug - specifically one of the driver's userspace API.
And these sort are old. I wrote a super [root] exploit for OS-9 which worked by tricking the GETSTAT call [equivalent of ioctl] for some driver to write its result to a random, unchecked place in memory. That was in 1992 / OS-9 2.4.
the root cause is in the kernel component that deals with memory inappropriately though. user space should be free to blast whatever into kernel space via system calls and ioctl and rely on the operating system to not crash, and also for the kernel to continue enforcing the rules of the operating system.
this is like saying "the root cause is Dan, who typed in a `drop tables. our solution is to take away Dans keyboard"
I have no doubt that you guys have great support, but since the MegaCLI utility that triggers the issue is a closed source third-party piece of software, I could imagine the process of getting Red Hat engineers to even reproduce the problem would probably take longer than just going in and fixing it themselves. I can imagine having hosts with dozens of VMs going down is a high-priority issue - OK, less so once they discovered they could work around it by disabling monitoring, and eventually by upgrading the monitoring utility, but at that point they were already close to a fix. (also: they didn't say they didn't report it to RH...)
On a more technical note, if the MegaRAID driver only does DMAs to/from 32-bit memory addresses, this could be an artificial performance limitation on systems with lots of RAM (like the 128GiB in the article) as it potentially means waiting for <4GiB physical memory buffers to become available, and copying the data. It's not clear from the article if the 32-bit-only restriction only applies to STP requests, or all I/O. As the STP stuff only seems to be used for monitoring, that's not a problem, but if all disk I/O uses bounce buffers, that's a bad thing.
I wonder if a hardware IOMMU could have caught this bug.
The situation was certainly bad with classic PCI - I have an nForce5-based motherboard whose PCI bus silently truncates 64-bit DMA buffer addresses to 32 bits. The onboard ethernet chip does in principle support 64-bit DMA, so I was getting all sorts of weird behaviour after upgrading my RAM from 2 to 6 GiB. The southbridge chip in question is now blacklisted for 64-bit DMA in the Linux kernel.
The 32-bit DMA cutoff isn't the only such constraint on buffer physical addresses of course; historically, ISA devices could only DMA to 20-bit addresses IIRC (or was it 24 bits?). I wrote an OS X driver for the virtio paravirtual ethernet adapter a while back[2] and its I/O buffers must be 4K page-aligned (low 12 bits 0), and its high bits are specified by a 32-bit number. This means you can only use buffers with physical addresses which fit in 44 bits. That's 16TiB, which are probably still a way off but could conceivably be reached not too far in the future. Current x86_64 CPUs certainly support 48 bit physical addresses in principle.
[1] That doesn't mean hardware bugs don't exist, but at least it's part of the basic spec now, as opposed to being tagged on later as for PCI.
We've talked about it before http://www.anchor.com.au/blog/2009/02/the-value-of-commercia...
I do have to wonder: did this bug all of a sudden spring up on you? Seems like the potential would have always been there on this platform. Could a change management process have lead you to find that "adding megaCLI monitoring to all KVM hosts" or "upgrading to megacli 8.01.06" was the cause of the problem? Just curious if you perceive a better CM or system inventory could have allowed you to deduce the problem area with a bit more efficiency?
Far as I can discern (I'm not a kernel hacker) that is the root cause. Can anyone with static code analysis experience chime in on whether it's possible to detect that kind of error by examining the source?