Show HN: A “living” Linux process with no memory
github.com
github.com
I've taken to calling such a process a "lich," because it's not quite a zombie - the parent can reap a zombie by calling wait(), but the lich has used magic to avoid true death.
That doesn't solve the kernel thread being stuck forever problem. I'm not sure what the fix is there... is it bad drivers with no timeout mechanism? I don't know how they'd do that unless the kernel IO is all async anyway. Just tearing down the kernel thread seems likely to leave the locks abandoned in bad states.
In the end, you really have little choice but to trust what the OS is telling you, and if the OS is lying for whatever reason, to fix the OS. As hard as that last bit may be as a solution, it still turns out to be the easiest, most-effective fix. And it may not be a perfect fix, either... but it'll still be the best.
Sounds like a hack.
So while I agree with the other posts that a supervisor can't recover from this state, it can be aware that it's happened. And it's quite common for supervisors to check for readiness and liveness beyond "well the OS says it's running, sounds good to me."
But TASK_KILLABLE is not used in most places it should be. Patching your (least) favorite driver to use TASK_KILLABLE could be a good entry point to contributing to the kernel.
This older, tangential HN discussion and the comments on LWN have a bit more info: https://news.ycombinator.com/item?id=18056946
That of course doesn't work if you're dealing with a filesystem that still uses TASK_UNINTERRUPTIBLE
[1] https://docs.microsoft.com/en-us/windows-hardware/drivers/dd...
[2] https://docs.microsoft.com/en-us/windows-hardware/drivers/ke...
That's why a process that wants to stay running has to provide some form of heartbeat API.
It was a java program that I killed -9. It died but was never reaped.
It prevented a software shutdown so I had to use the ol' STOP-A
Can a malicious process use this to avoid being killed
A malicious process can't do useful work - as soon as the kernel gets unstuck, the signal will get delivered to the thread and kill it.
This is the profit.
While job searching I went on an interview where I got asked why I did a side gig listed on my resumee. "For friend or for money?" It was neither and I said "I don't know, because programming is fun?"
I got hired.
Why are you learning how to create linux processes with no memory? Why are you hacking overloading into Python? Why are you writing programs for fun? To dig unnecessarily deep and force your way into programming's secrets.
So, for money.
Everything in between made it seem like making money my No. 1 focus was necessary, though. A lifetime of being manipulated and pressured. It all gets better when you can finally pull back the curtain seeing what's really there. :)
In other words : you can't derive a motivation from an action an individual does if it's something that they're forced to do so socially or legally.
Excerpt:
"There are advantages and disadvantages to each type of sleep. Interruptible sleeps enable faster response to signals, but they make the programming harder. Kernel code which uses interruptible sleeps must always check to see whether it woke up as a result of a signal, and, if so, clean up whatever it was doing and return -EINTR back to user space.
The user-space side, too, must realize that a system call was interrupted and respond accordingly; not all user-space programmers are known for their diligence in this regard.
Making a sleep uninterruptible eliminates these problems, but at the cost of being, well, uninterruptible. If the expected wakeup event does not materialize, the process will wait forever and there is usually nothing that anybody can do about it short of rebooting the system.
This is the source of the dreaded, unkillable process which is shown to be in the "D" state by ps.
Given the highly obnoxious nature of unkillable processes, one would think that interruptible sleeps should be used whenever possible. The problem with that idea is that, in many cases, the introduction of interruptible sleeps is likely to lead to application bugs.
As recently noted by Alan Cox:
Unix tradition (and thus almost all applications) believe file store writes to be non signal interruptible. It would not be safe or practical to change that guarantee."
That's the whyness to all of this...
> If the process is in uninterruptible sleep then the process can’t be interrupted, which will cause the strace process itself to hang forever. Remarkably, it appears that the ptrace(2) system call is itself uninterruptible, which means that if this happens you may not be able to kill the strace process!
Everything about this is broken and makes no sense. The Night Watch is as relevant as ever. (https://www.usenix.org/system/files/1311_05-08_mickens.pdf)
$ pip2 install fuse
Collecting fuse
Downloading https://files.pythonhosted.org/packages/c3/f6/82777531d0dd0fa1d1b509258873f4b48e1ec702dcf0258214fafb474895/fuse-0.1.3.tar.gz
ERROR: Packages installed from PyPI cannot depend on packages which are not also hosted on PyPI.
fuse depends on gunpowder@ git+https://github.com/funkey/gunpowder@721718b6569b47a2f5d5d6633c76c85f779e25c7It may already be installed!
$ python2 fs.py x
Traceback (most recent call last):
File "fs.py", line 8, in <module>
class fs(fuse.Operations):
AttributeError: 'module' object has no attribute 'Operations'
$ python2
Python 2.7.17 (default, Nov 7 2019, 10:07:09)
[GCC 7.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import fuse
>>> fuse.__version__
'0.2.1 from fuse import fuseI've found it very accessible. Largely, the way that they teach is to describe a system based on a set of assumptions, then slowly relax each of the assumptions one by one until you reach an example of a real system. That style of teaching really works for me.
Uninterruptible process sleep is covered in section 22.3. Threads, in chapters 29–33. Signals, chapters 20–22. The proc file system in section 12.1. Memory mappings, chapters 49–50.
Uninterruptible sleep, signals, memory mappings etc have not changed fundamentally.
Namespaces and cgroups are new concepts in Linux, they won't be covered. IIRC user namespaces appeared in 3.14. New cgroup hierarchy got somehow usable much later than that.
The new concepts are needed for containers. Not at all relevant for the original article here.
Not sure why people care what it's written in. I'm pretty sure people wouldn't complain if this was a blog post written in english rather than a program written in C and python2.
Fuse is restricted; probably for reasons like this.
Basically, what’s happening is that the FUSE daemon which is the one handling the FUSE requests has /dev/fuse open. There is a thread in that FUSE daemon. let’s call the FUSE daemon P10, and the thread P11.
P10 wires up a FUSE filesystem on /tmp, it opens /dev/fuse with FD5 P11 enacts an uninterruptible (blocking operation) on /tmp/foo, called OP1, and /tmp/foo is FD6. P10 reads the operation (OP1) from /dev/fuse, so now OP1 is in userspace ----The Pid 1 of the namespace is killed--
P10 just terminated, and cannot make progress. It will never respond to OP1. FD5 and FD6 remain open, because P11 is in uninterruptible sleep. P11 is in an uninterruptible disk sleep waiting for OP1 to respond, and the fuse connection never aborts.
FUSE abortion doesn’t kick in because the FD of `/dev/fuse` that P10 originally opened as the FUSE daemon under FD5 will not be closed until all threads as part of the process are terminated. The mount namespace wont be torn down until P11 is torn down. P11 will never be torn down because it’s waiting for someone to do something.
[0] Disclaimer: may take slightly longer than heat death of current universe.
[1] Alternative solution: https://m.xkcd.com/1266/
In terms of the actual unmap call, the payload is in the munmap call, which is a syscall, and if you do have to set up the registers for that call, Go does have its assembler support which should be able to do that for you. You should be able to write Go source code that would perform that call, although you're going to be poking into some corners of Go most programmers never have to get into. You can also get to fuse, which in my experience is great at producing uninterruptible sleeps even when you're not trying to do it on purpose....
However, in terms of reliability and the long term, you have the problem that you're not the only thread in the program the way a C program can count on that, and if any of those other threads wake up and their memory is missing, it'll probably result in the process dying. You can do a couple things to try to avoid that a little, like pinning yourself to an OS thread and running the program in one CPU mode, but especially on the latest version of Go where they've implemented true pre-emption, you can't keep the runtime from running by just hogging the execution thread anymore.
So I suspect that as long as you win the race to unmap everything before the runtime wakes up (which may also require you to call sysmap through the assembly code directly, rather than using the syscall libraries as I'm pretty sure those notify the scheduler of what you're doing), you may be able to briefly be a process with no RAM mapped, but in human terms it won't be long before the runtime wakes up to do something, anything, and crash the process as a result. There's no way to avoid that in Go. You won't be able to have a process just sitting there indefinitely with no mapped RAM that you can admire and treasure and hand off to your children as part of their inheritance.
Note that if you link against libc I think it would be legal for it to create a thread behind your back, so even in this case you're not quite safe.
The program gets a list of all the allocated pages first, then creates one more for the JIT code (and a copy of the list) which is unmapped as the last thing it does. There really is no other way of doing it.