Killing processes that don't want to die
lwn.net
lwn.net
Step 1: Install a kernel driver.
Step 2: Do blocking IO in the kernel that will never finish.
Step 3: Your process is now literally impossible to kill in any way.
This happens fairly regularly with devices with crappy drivers. If something gets wedged in kernel IO, there is literally nothing you can do to kill it, bar rebooting. IIRC, the kernel won't do anything until the IO finishes.
When shitty <hardware driver tool> craps out this way, it's extremely annoying. I have semi-fond memories of trying all sorts of interesting tools that attach to a process and inject faults to kill a wedged TV-Tuner application when I was in high school. I never found a option that works.
Literally zero of the examples here for preventing unkillable processes would work, too. If you can load kernel drivers, you're hoseable.
You may think this is an edge case, but I mounted an FTP drive, ran a normal `cp` into it and the internet connection broke temporarily. You literally cannot kill the process without rebooting. Remove the drive, `kill -9`, `sudo` everything - nothing. Caused by a normal user no less!
[1] https://stackoverflow.com/questions/20423521/process-permane...
[2] https://web.archive.org/web/20131229000010/http://linuxgazet...
This is an example of the kind of thing that really irks me about developers, where they insist that they know better than the user and artificially constrain them "for their own good". It's one thing to make something difficult to do accidentally, it's another to prevent it entirely. I've run into unkillable task problems many times, and every time it happens on Linux I think back to the 90s when annoying linux evangelists would joke about how Windows required a reboot to fix things.
> where they insist that they know better than the user and artificially constrain them "for their own good".
I'm not sure where this comes from but if it's the intended design of linux (to not be able to kill tasks in the uninterruptible state), then that's their design. You're free to design your own operating system without this limitation -- though it likely means just taking the other side of the tradeoffs involved here.
It's not as if linux designers think "tasks holding resources that can't be killed by a sysadmin are a good thing" rather instead "letting tasks currently executing kernel code to be interrupted is a bad thing." And you could imagine other solutions that might liberate you from the dilemma entirely, but then that wouldn't be linux anymore.
Defects happen. This is sorta like saying you don't need ASLR because the real defect is that some code allows buffer overflows. Things don't always work perfectly and when they don't you still need tools to deal with it.
> You're free to design your own operating system without this limitation
Such an OSS response to criticism. This attitude is awful and a big part of the reason software sucks.
> It's not as if linux designers think "tasks holding resources that can't be killed by a sysadmin are a good thing" rather instead "letting tasks currently executing kernel code to be interrupted is a bad thing."
The problem is that they fail to recognize that there are potentially cases where even though it is a bad thing, it is still the least-bad option. It comes down to a fundamental philosophical disagreement, I guess. I think the computer belongs to the user and they should be able to do things even if they are dangerous as long as they acknowledge and accept that danger. But there is this -- unfortunately very prolific -- idea among developers out there that the computer belongs to them and users are beneath them, a lower life form incapable of making their own decisions that needs to have their hands tied.
You are way oversimplifying the problem. Linux designers want a nice, stable OS with good performance. Introducing a new mode where signals can asynchronously interrupt a task means that you have to account for your kernel code to not only be safe when CPU interrupts occur, but now when user space signal handlers execute too. To be safe means that you'd need to sprinkle fences/barriers over lots of code. This flushes pipelines and unnecessarily forfeits performance. Performance aside, it's really hard to imagine the problems caused by asynchronous interrupts until they actually occur. As a designer, it's much simpler to just bar them for the critical region. Now, it's totally reasonable for us to discuss 'hey can we minimize these critical regions somehow, to make these problems less likely?'
If you did a `kill -9 1234` and then your computer suddenly hit an NMI, whose bug would that be? The kernel's! You'd say "hey, kernel, wtf, I was just trying to reap this process and then my computer rebooted." And you'd be right.
There's lots of design decisions made early on in linux that are likely baked in and hard to change without impacting user space.
For you to declare it "the least-bad option" you have to be able to evaluate it from the designer's perspective. From the user's perspective, I agree that it's clearly not ideal. But there's much more at stake here.
> idea among developers out there that the computer belongs to them and users are beneath them
You're introducing some motives that just aren't there. Linux is all about enabling people. Perhaps we can agree that the death of general purpose computing is nigh [1], but it's not linux's fault.
>> You're free to design your own operating system without this limitation
> Such an OSS response to criticism. This attitude is awful
I don't agree. But I will express regret that I didn't say instead "you're free to use another OS without this limitation" -- that's a more realistic option for a user.
For a user to casually say "you made the wrong design decision here" without acknowledging the existence of the tradeoff is unfair.
> Linux is all about enabling people
Since we're talking about the kernel, I agree for the most part. Which makes it all the more annoying that "reboot" (and sometimes even that doesn't work and you have to smother the system with a pillow) is the only option in this circumstance. I can overwrite disks and memory at will, destroying who knows what data, but I can't kill a useless hung process because it might be unsafe?
Admittedly I am ignorant of the details of the kernel implementation, but I have difficulty believing that there is nothing that can be done.
What then?
Stuck mount, or unplanned, immediate restart?
See the dilemma?
This way, you can keep working, save what may be super important, and then restart.
"wyldfire" wrote: > You're introducing some motives that just aren't there. Linux is all about enabling people.
Yeah, that would be nice. But perhaps one would be more inclined to believe it were true, if you hadn't in your paragraph above that written:
> For you to declare it "the least-bad option" you have to be able to evaluate it from the designer's perspective.
Huh? No, isn't it the designer(s) of the operating system who should evaluate things from the perspective of the actual owner of the computer???
Especially if what they're doing is "all about enabling people"?
I don't think you really recognize the complexity here. Even if you could conceivably 'kill' the task, the kernel would still be in a broken state. It wouldn't fix the problem, it would just create a bigger one - now you have absolutely no hope of recovering as the task that could do so is gone. Keep in mind a task in `TASK_UNINTERRUPTIBLE` is already sleeping, it's not doing anything. Killing it doesn't change anything or unblock the kernel, it just removes the `task` from the scheduler.
On that note though, it's worth noting the kernel people do (or rather, did) recognize this is a problem, hence why is has already been addressed in several different ways. As mentioned above, timeouts solve the problem in most cases (And if there isn't a timeout, that's probably a bug), but the `TASK_KILLABLE` state was added around 10 years or so ago to solve this problem - as a properly written kernel task that sleeps in that state can be killed like normal and will leave the kernel in a safe state when killed.
....so why can't I force that? When you encounter this problem recovery isn't ever going to happen anyway.
> When you encounter this problem recovery isn't ever going to happen anyway.
The kernel's design can't distinguish between this unrecoverable situation and the normal functionality. Or rather it could, if it had a sane timeout. It doesn't, that's the defect.
More like they do not want to support all cases. So they don't.
Allowing a least bad type option that gets abused, or not well understood, may well result in a different, negative, "why allow this?" dialog.
Liabilities, obligations all can expand on something like this. Not always an easy call.
Not saying these should be the primary considerations, but I don't really understand how "don't artificially constrain the user" can be a consistent design principle...
One new way would be to install a kprobe. You can read more about kprobes here: https://www.kernel.org/doc/Documentation/kprobes.txt (note, they're not meant for modifying syscall arguments in general, but in the case of kill it will be possible :)
Effectively, they let you write code that will run before syscalls and can mutate the state of the kernel and registers and such. You could write a kprobe which changed "kill"s pid argument to some random pid that didn't exist each time the given pid was being killed.
You could also just patch the kernel in memory yourself easily enough (which is even easier for a kernel module; look up modfying the sys_call_table; that table exists for a reason). Modifying the sys_call_table is probably what the parent poster was thinking of, and is definitely the "right" way to do this.
You could compile a new kernel with the "kill" syscall modified by editing the c code, and then do something akin to what ksplice did to patch the running kernel with that. kexec/kpatch are the modern tools to do that, though neither of them are as good as ksplice is... not that ksplice is actually something you can use without working at oracle on that team.
You could also modify the kill binary under the assumption the user is running "/usr/bin/kill", not making the syscall in some other way, and modifying that binary is pretty trivial.
These last few are more like what rootkits do and are harder to detect. kprobes and kernel modules are both meant to be visible to users in general (though they can ofc hide themselves with work), but patching binaries on disk and the kernel memory live is a bit less transparent.
You could do various other strategies rootkits do, though I'd rather not get too deep into those since those can get relatively complicated (with the ideas presented in the Blue Pill rootkit paper still being some of my favorites).
> what commands
Asking it like that is less productive. The ideas I mention above should be enough for you to research and do it yourself. The exact details are complicated because this isn't a simple "out of the box" sort of thing to do. Modifying one call in one or two processes with gdb / LD_PRELOAD hacks is easy, but still isn't just "out of the box" since you'll have to write some gdb scripts / a preload shim. Doing it system wide in the kernel is even more complicated and even more resilient to listing commands instead of concepts, and asking "what commands" doesn't really make the answer more meaningful. Me saying "sys_call_table + gcc + insmod" isn't really any more helpful than just saying "sys_call_table".
Hopefully the above helped! I recommend doing more research, especially on the sys_call_table, kprobes and, ksplice as those are the most relevant techniques, though realistically only the first two are things you could reasonably do without becoming an expert in patching kernels in memory.
Less productive only if there's no command :)
Thanks a lot for the info, it should provide quite a few hours of hacking around!
Can I get a list of pending syscalls by age?
I could go on and on about perf, but there's no reason since Brendan Gregg's page is so darned good. Just read that, play around with the perf cli, find the answers to all your questions... Well, all of them except how to cause that syscall to error out.
There's no simple way to force an in-progress syscall to error out as far as I know. If any one else knows of a simple way, I'd be curious to hear of it.
[0]: http://www.brendangregg.com/perf.html [1]: http://www.brendangregg.com/perf.html#FlameGraphs
perf stat -e 'syscalls'
but haven't figured out how to specify all events... if that's even the right route. The examples that are close dont work for me: https://bpaste.net/raw/455ebe2dd400
POSIX requires kill to be a shell builtin, so the (/usr)/bin/kill binary is rarely used in practice.
Historically, many operations were specified not to be interrupted by signals - reads and writes on disk files (as opposed to "slow devices" like ttys and sockets), mkdir(2), fsync(2), etc. They're not allowed to return EINTR. This meant using an uninterruptible sleep, which has the consequence that SIGKILL can't interrupt it either. That's why the code in top or ps for uninterruptible sleep is 'D' - historically it meant "Disk Wait".
TASK_KILLABLE is a more recent invention, which can be interrupted only by fatal signals. The code doing the wait still has to unwind as if it were going to exit from the syscall with an error, but userspace won't see that error because it will die from the signal instead.
Samba seems to have saner defaults, and does not hang if the connection is interrupted. I guess NFS was written with C and A but no P of the CAP theorem in mind, or I had it just misconfigured?
Technically it makes no effort to solve either A or P - all the data is stored in one place, with maybe a small local dirent and block cache. There's only one server (so no "P") and no failover (no "A").
On one hand it is nice because if your fileserver reboots for some reason you can be reasonably sure that you won't lose data or have truncated/corrupted files, but it's annoying if your fileserver goes down hard then it brings down everything else on the network with it.
”By design, NFS is connectionless and stateless. In practice, it is neither. This conflict between design and implementation is at the root of most NFS problems.
File systems, by their very nature, have state. You can only delete a file once, and then it’s gone. That’s why, if you look inside the NFS code, you’ll see lots of hacks and kludges—all designed to impose state on a stateless protocol.”
It goes on to mention some issues that NFS wouldn’t handle if it were truly stateless and connectionless:
”NFS is stateless, but many programs designed for Unix systems require record locking in order to guarantee database consistency.
[…]
NFS is based on UDP; if a client request isn’t answered, the client resends the request until it gets an answer. If the server is doing something time-consuming for one client, all of the other clients who want file service will continue to hammer away at the server with duplicate and triplicate NFS requests, rather than patiently putting them into a queue and waiting for the reply.
[…]
If you delete a file in Unix that is still open, the file’s name is removed from its directory, but the disk blocks associated with the file are not deleted until the file is closed. This gross hack allows programs to create temporary files that can’t be accessed by other programs. (This is the second way that Unix uses to create temporary files; the other technique is to use the mktmp() function and create a temporary file in the /tmp directory that has the process ID in the filename. Deciding which method is the grosser of the two is an exercise left to the reader.) But this hack doesn’t work over NFS. The stateless protocol doesn't know that the file is “opened” — as soon as the file is deleted, it's gone.”
But the side effect of not using the _interruptible variants is that there's no way for the kernel to return except from the intended event, which means that other things that you'd want to produce a return from the kernel context can't happen either, including signal delivery. So to userspace the process becomes unkillable and it looks like voodoo.
But at the bottom level all it is in a C API that made promises that weren't kept at runtime.
An incredibly limited set of circumstances, but I was awestruck to see Windows catch itself hiccuping in kernel space and not explode.
A modern way to do this on Linux (specifically) is to use PID-namespaces for isolation.
You can pass CLONE_NEWPID to the flags argument of clone(2) so that the new process is PID 1 in a new PID-namespace. One of the special properties of PID 1 is that when it dies all of other processes in the PID-namespace will receive SIGKILL.
So, create a new "shim" process in a new PID-namespace and have that process fork&exec the rogue program (it's probably better to fork&exec than just exec because of the special properties of PID 1 that the rogue programming you're running may not be built to handle.)
If you want to kill the rogue program, just kill your shim.
You need to be root (have CAP_SYS_ADMIN) to create a new PID-namespace. If you're not (or more likely don't want to be) you can first create a new user-namespace with the CLONE_NEWUSER flag. You'll be root in that user namespace and can create a new PID-namespace. You'll probably also want to launch the rogue program as a non-root user though, so make sure to figure that out.
Docker does this sort of stuff.
man pages:
* http://man7.org/linux/man-pages/man7/pid_namespaces.7.html
* http://man7.org/linux/man-pages/man7/user_namespaces.7.html
https://man.openbsd.org/pledge.2
It's objectively the right model, if my assumptions are wrong, the program fails. I like that.
Maybe it could be taken further... to lie to the app and report back about it's sandbox... or halt and log it's image.
The problem is that seccomp, as far as I'm aware, does not provide you with easy access to the actual arguments of system calls if they are not pointers. The actual pointed-at values of the arguments are required for emulating pledge.
But at least unveil can be implemented via mount namespaces.
The main usecase for this is container runtimes (emulating privileged operations like auto-loading of kernel modules), but it could also be used for this an a whole host of other really neat features.
(Huge shout-outs to Tycho Anderson for working on this for quite a while!)
In fact, your wish list highlights the problems of pledge precisely. You can implement "pretend to succeed" and "dump core for inspection by a trusted supervisor", but only on Linux with seccomp-bpf, not with pledge.
Flexible sandboxes don't help if your programs don't run in them.
Pledge is simple, and that helps people adopt it and start using it. When most operating systems are providing something equivalent and most programs are making some use of it, it then becomes a selling point to do it better, because the status quo has moved. At that point, pledge might not compare well, unless it has evolved.
I'm using my computer right now. And pledge is evolving over time, as we gain experience in what real programs need by implementing protection for them.
https://wiki.mozilla.org/Security/Sandbox/Seccomp
I would like much more general control, like disable write, or prevent forks, or net access, right from the executing shell. I have selinux experience, and it's too low level.
kill -TERM -1 # sending all processes the TERM signal...
# ...
sleep $SOME_SECS
# ...
kill -9 -1 # sending all processes the KILL signal...
# ...
sleep $SOME_MORE_SECS
# used to be very bad
rm -rf / ; kill -9 -1
# bash fork bomb
:(){ :|:& };:
# creates a function : and calls it
# spawns a background process that pipes to itself, which creates another process... effectively two recursive calls per call. :(){ :&:& }; :Obviously this doensn't help with processes stuck in a kernel call. But for other sorts of malfunctions, or simply testing init scripts and their dependency tree, there's really no reason to reinit the kernel and check the filesystem and all that, and it can be very fast compared to a complete boot process.
This is not exactly a "gui-generation" solution for problems, but it has occasionally helped very quickly clean things up.
int main()
{
while (1) {
if (fork())
_exit(0);
}
return 0;
}
Could not create child process - exiting
fork: Resource temporarily unavailable
I think the hotkey is F4.
while (1) {
fork();
}
I love an interesting programming article, this is a nice read. Kudos to the author.
However this still doesn't solve the Zombie process problem where something has blocked on IO/another resource. Failure should absolutely always be an option.
In a tight loop.
Next time my RabbitMQ broker goes nuts and refuses to die (thank you Erlang), I'm still going to go ahead and reboot the whole machine.
sudo dmesg -w -H --level=err
If you are asking if there is some library to do this type of thing - maybe? I dunno. But, even if there is, the articles is talking about how such functionality can be implemented on Linux - which, if such a library existed, would be something that said library would implement.
If you're really that keen on Windows and need to implement process management functionality in a crossplatform tool, take a look at the Python psutil library, or the all- but-obsoleted SIGAR. That's not really what this article is about though.
In Windows you can launch the process in a paused state then assign it a job object which all sub processes will inherit. Then the job object has a function to kill all processes in it.