"Doors" in Solaris: Lightweight RPC Using File Descriptors (1996)
kohala.com
kohala.com
Doors at the end of the day aren't message passing based RPC. There is no door_recv(2) syscall or equivalent nor any way for a thread pool in the callee to wait for requests.
Doors at the end of the day are a control transfer primitive. In a very real sense the calling thread is simply transferred to the callee's address space and continues execution there until a door_return(2) syscall transfers it back into the caller address_space.
It truly is a 'door' into another address space.
This is most similar to some of the CPU control transfer primitives. It's most like task gate style constructs like seen on 286/i432/mill CPUs. Arguably it's kind of like the system call instruction itself too, transferring execution directly to another address space/context.
maybe https://docs.oracle.com/cd/E88353_01/html/E37843/door-create... is where i should start reading...
Doors Implementation
[...] the default server threads that are created in response to incoming door requests on the server.
A new synchronization object, called a shuttle [...] allowing a direct scheduler hand-off operation between two threads.
[...] a shuttle marks the current thread as sleeping, marks the server thread as running, and passes control directly to the server thread.
Server Threads
Server threads are normally created on demand by the doors library [...]
The first server thread is created automatically when the server issues a door create call.
[...] Once created, a server thread will place itself in a pool of available threads and wait for a door invocation.
> There is no door_recv(2) syscall or equivalent nor any way for a thread pool in the callee to wait for requests
It says the thread pool on the server is created by the doors library when the server creates a door. So the process of receiving and processing requests would be carried out internally within the doors library, so there would be no need for the server application to have an API for the server application to accept requests, it's handled by the library.
At least that's what is described in the link, AFAIKS. It's only a conceptual door, underneath it is implemented in some message passing style maybe with some extra sugar for performance and process control nicety with this "shuddle" thing for making the requests.
man pages like https://docs.oracle.com/cd/E88353_01/html/E37843/door-server... reference literally zero man pages in section 2, so i wonder if there is in fact a door_recv system call and it just isn't documented?
but yeah it sure seems like there's a thread pool of server threads (full-fledged posix threads with their own signal masks and everything) that sit around waiting for door calls
> The door_server_create() function allows control over the creation of server threads needed for door invocations. The procedure create_proc is called every time the available server thread pool is depleted. In the case of private server pools associated with a door (see the DOOR_PRIVATE attribute in door_create()), information on which pool is depleted is passed to the create function in the form of a door_info_t structure. The di_proc and di_data members of the door_info_t structure can be used as a door identifier associated with the depleted pool. The create_proc procedure may limit the number of server threads created and may also create server threads with appropriate attributes (stack size, thread-specific data, POSIX thread cancellation, signal mask, scheduling attributes, and so forth) for use with door invocations.
<https://docs.oracle.com/cd/E88353_01/html/E37843/door-server...>
apparently things like door_create() survived into opensolaris and so they are presumably open source now? even if under the cddl
<https://www.unix.com/man-page/opensolaris/3c/door_create/>
/me git clone https://github.com/kofemann/opensolaris
jesus fuck, 1.4 gigabytes? fuck you very much fw_lpe11002.h
okay so usr/src/lib/libc/port/threads/door_calls.c says the 'raw system call interfaces' are __door_create, __door_return, __door_ucred, __door_unref, and __door_unbind, which, yes, do seem to be undocumented. they seem to have been renamed in current illumos https://github.com/illumos/illumos-gate/blob/master/usr/src/...
unfortunately it's not obvious to me how to find the kernel implementation of the system call here, which would seem to be almost the only resort when it isn't documented? i guess i can look at how it's used
__door_create in particular is called with the function pointer and the cookie, and that's all door_create_cmn does with the function pointer; it doesn't, for example, stash it in a struct so that a function internal to door_calls.c can call it in a loop after blocking on some sort of __door_recv() syscall (which, as i said, doesn't exist)
it does have a struct privdoor_data under some circumstances; it just doesn't contain the callback
i don't know, i've skimmed all of door_calls.c and am still not clear on how these threads wait to be invoked
aha, the kernel implementation is in usr/src/uts/common/fs/doorfs/door_sys.c. door_create_common stashes the function pointer in the door_pc of a door_node_t. then door_server_dispatch builds a new thread stack and stashes the door_pc on it as the di_proc of a door_info_t starting at line 1284: https://github.com/kofemann/opensolaris/blob/master/usr/src/...
this seems to be a structure with layout shared between the kernel and userspace (mentioned in the man page i quoted above): https://github.com/illumos/illumos-gate/blob/master/usr/src/...
...but then door_calls.c never uses di_proc! so i'm still mystified as to how the callback function you pass to door_create ever gets called
probably one hour diving into solaris source code is enough for me for this morning, though it's very pleasantly formatted and cleanly structured and greppable. does anybody else know how this works and how they got those awesome numbers? is door_call web scale?
The aha moment was when I realized that the door_return(2) syscall is how threads yield and wait to service the next request. In retrospect it make sense, but I didn't see it until I tried to figure out how a user space thread polled for requests--a thread first calls door_bind, which associates it to the private pool, and then calls door_return with empty arguments to wait for the initial call. (See example door_bind at https://github.com/illumos/illumos-gate/blob/4a38094/usr/src... and door_return at https://github.com/illumos/illumos-gate/blob/4a38094/usr/src...)
One of the confusing aspects for me was that there's both a global pool of threads and "private" pool of threads. By default threads are pulled from the global pool to service requests, but if you specify DOOR_PRIVATE to door_create, it uses the private pool bound to the door.
AFAICT, this is conceptually a typical worker thread pooling implementation, with condition variables, etc for waking and signaling. (See door_get_server at https://github.com/illumos/illumos-gate/blob/915894e/usr/src...) Context switching does seem to be optimized so door_call doesn't need to bounce through the system scheduler. (See shuttle_resume at https://github.com/illumos/illumos-gate/blob/2d6eb4a/usr/src...) And door_call/door_return data is copied across caller and callee address spaces like you'd expect, except the door_return magic permits the kernel to copy the data to the stack, adjusting the stack pointer before resuming the thread and resetting it when it returns. (See door.S above and door_args at https://github.com/illumos/illumos-gate/blob/915894e/usr/src...)
This works just as one might expect: no real magic except for the direct thread-thread context switching. But that's a similar capability provided by user space scheduler activations in NetBSD, or the switchto proposal for Linux. The nomenclature is just different.
It's a slightly different story for in-kernel doors, but that's not surprising, either, and there's nothing surprsing there, AFAICT (but I didn't trace it as much).
Thanks for finding those source code links. I cloned the repo and started from there, grep'ing the whole tree to find the user space wrappers, etc.
so door_return (or, in opensolaris, __door_return) is the elusive door_recv syscall, eh?
(i'll follow the rest of your links later to read the code)
i wonder if spring (which was never released afaik) used a different implementation, maybe a more efficient one
this is strongly reminiscent of ipc in liedtke's l4 (which doesn't implicitly create threads) or in keykos (which only has one thread in a domain)
thank you so much for figuring this out!
The more I read, the more Doors seems pretty clever. Yes, the secret sauce is very much like scheduler activations or Google's switchto, but you can pass data at the same time, making it useful across processes, not just threads sharing an address space. And you can't get the performance benefit merely by using AF_UNIX sockets because the door_call and door_return have an association for the pendency of the call. The tricky part from an API perspective isn't switching the time slice to the server thread, it's switching it back to the client thread on the return.
Microkernels can do this, of course, but figuring out a set of primitives and an API that fit into the Unix model without introducing anything too foreign, like Mach Ports in macOS, does take some consideration. Now that Linux has process file descriptors, maybe they could be used as an ancillary control message token to get the round-trip time slice and data delivery over AF_UNIX sockets. And memfd objects could be used for messages larger than the receiver's buffer, which seems to be a niche case, anyhow. Or maybe that's over complicating things.
obviously this is the opposite extreme from 'fit into the unix model without introducing anything too foreign' ;) and it may turn out that it's not as good an idea as it seems once i try it. but while some aspects of the unix model are very good, i suspect others were more meritorious on non-networked pdp-11
with respect to starting up a thread to run a callback registered for a message port, i feel like it could be done with a great deal less code than this, though maybe i'm being too optimistic about keeping messy real-world complexity out. in einkornix, which i've only tested as a user-level threading library under linux in emulation, my function for starting up such a thread is einspawn(), which is called (like door_create()) with a callback function and a userdata argument. __door_create() also takes a flags argument; einspawn() instead takes a pointer to the task struct it will finish initializing. it's ten instructions long: http://canonical.org/~kragen/sw/dev3/einkornix.S
The client’s stack isn’t present in the server’s address space: the server needs a pool of threads so that a door call does not need to allocate memory for the server stack.
Together with the other points about marking the current thread sleeping and passing control to the server thread, and about server threads waiting for an invocation, I think what is described is pretty clearly the server threads are being used to execute code on the server on behalf of the client request.
So, yes. The documentation in right, the calling thread is paused while a different one is used one the other side of the door.
Your phrasing is misleading. A door is a scheduler operation. What would it even mean for a thread to go from a caller to a callee? They are different processes with completely different contexts on the system. Different owners, different address spaces, etc. What the door is doing is passing time on a CPU core from the calling thread to the receiving thread and back.
This is true, but isomorphic to RPC, so I really don't think I understand the distinction you're trying to draw. A "procedure call" halts the "caller", passes "arguments" into the call, and "returns a value" when it completes. And that's what a door call does.
Nothing about "Remote Procedure Calls" requires thread pools or a recv() system call, or an idea of "waiting for a request". Those are all just implementation details of existing RPC mechanisms implemented without a send+recv call like doors.
Nor FWIW are doors alone in this space. Android Binder is effectively the same metaphor, just with a more opinionated idea of how the kernel should manage the discovery and dispatch layer (Solaris just gives you a descriptor you can call and lets userspace figure it out).
If it were a message passing system, you would be able to save the message to disk and inspect it later, or transmit the message over the network, etc.
If so, then I retreat to the argument above: there is nothing about "Remote Procedure Call" as a metaphor that requires "message passing". That, again, is just an implementation detail of other RPC mechanisms that don't implement the call as directly as doors (or binder) do.
No. This is just colloquial usage of the term “message passing”.
Yes, you can use doors to pass messages. That is not the only thing you can do.
> If so, then I retreat to the argument above: there is nothing about "Remote Procedure Call" as a metaphor that requires "message passing".
Yeah. Everyone else here agrees with that. The original comment you replied to said, “Doors at the end of the day aren't message passing based RPC.” This absolutely indicates that the poster agrees with you on that point—that not all forms of RPC are based on a message passing system. There are other forms of RPC, and this is one of them.
Ultimately, I think you can conceive of “message passing” broadly enough that, well, everything is message passing. What is a CPU register but something that receives a message and later re-transmits it? I would rather think of the colloquial use of “message passing”.
Likewise, you can probably dig into doors and find some message system somewhere in the implementation. Or at least, you’ll find something you can call a message.
This works great as long as the server thread isn't itself doing slow I/O. The moment the server thread does slow I/O you want to get back to async I/O, and the whole point of doors is that it's synchronous because only by being synchronous do you get to transfer priority and CPU quantum from the client to the server. For async RPC-like schemes you cannot transfer priority and CPU quantum to the server because the client will continue executing with that same CPU quantum (that being implied by "async").
Also, this really only works locally. For remote services the server could get priority metadata from the client's authentication credentials, but that's it.
The Linux server revolution was still a few years away (at least it was for us – perhaps we were behind the curve), so our webservers were all Sun Enterprise machines running Solaris.
We decided to use a server process that had an in-memory representation of the active users, which we would query using doors from a custom Apache module. (I had read about doors and thought they sounded cool, but neither of us had used them before.)
It worked brilliantly and was extremely fast, though in the end it was never used in production because of changing business priorities.
My first project as a junior was to deploy Solaris Zones to replace a bunch of old SPARC machines. It's such a great tech and was a fun project.
Personally I think its more fair to say that Linux was behind the times because containerisation and virtualisation were common concepts in UNIXes long before they became Linux staples.
We used it for our HP-UX customers in telecoms, using our CRM software stack, based on Apache/Tcl/C.
I have a lot of admiration for what Docker dared to do-- to really think differently about the problem in a way which changed application deployment for everyone.
Also I can tell you at the time that we were not especially concerned about HP or IBM's solutions in this space; nor did we face those container solutions competitively in any sales situation that I can recall. This tech was fielded in the wake of the dot-com blowout-- and customers had huge estates of servers often at comically low utilization. So this was a good opportunity for Sun to say "We are aligned with your desire to get maximum value out of the hardware you already have."
It's a blast to see this come up from time to time on HN, thanks.
Still a nice paper!
> to really think differently about the problem in a way which changed application deployment for everyone.
Could that also have been done with Zones? In terms of the developer experience?
Seems to me Docker just thought about the develop-and-deploy pipeline differently.
Zones were awful.
Solaris had all of these great ideas, but the OS was an absolute pain in the ass to admin. And I say that from both sides of the SWAN firewall.
What on Earth for? Solaris has DTrace.
Note that Server/Client refer to threads on the same machine.
While I can see performance benefits of this approach, over traditional IPC (sockets, shared memory), this "opens the door" for potentially worse concurrency headaches you have with threads you spawn and control yourself.
Has anyone here hands-on experience with these and can comment on how well this worked in practice?
With a "door", a client is elevated to the "server" mode.
A quick look through BeOS and Android Binder-related APIs will quickly show how Android side is derived from it (through OpenBinder, which was for a time going to be used in next Palm system based on Linux, at least one of them)
Will binder ever make it into mainline Linux?
You can of course build your own use (depending on what exactly you want to do, you might end up writing your own userland instead of using androids)
Sharing of scheduler slice is an even older idea, AFAIK, and technically something already done whenever you call a kernel (it's not a context switch to a separate process, it's a switch to different address space but running in the same scheduler thread)
The "endpoint" is set up via door_create(); the client connects by opening it (or receiving the open fd in other ways), and make the request by door_call(). The service sends its response by door_return().
Except that the "handover" between client and service is inline and synchronous, "nothing ever sleeps" in the process. The service needn't listen for and accept connections. The operating system "transfers" execution directly - context switches to the service, runs the door function, context switches to the client on return. The "normal" scheduling (where the server/client sleeps, becomes runnable from pending I/O and is eventually selected by the scheduler) is bypassed here and latency is lower.
Purely functionality-wise, there's nothing you can do with doors that you couldn't do with a (private) protocol across pipes, sockets, HTTP connections. You "simply" use a faster/lower-latency mechanism.
(I actually like the "task gate" comparison another poster made, though doors do not require a hardware-assisted context switch)
The SPARC context switch code, to me, always was very complex. The hardware had so much "sharing" (the register window set could split to multiple owners, so would the TSB/TLB, and the "MMU" was elaborate software in sparcv9 amyway). SPARC's achilles heel always were the "spills" - needless register window (and other cpu state) to/from memory. I'm kinda still curious from a "historical" point of view - thanks!
Switching from task A to task B to service a "fast path" call AFAIK (have no access to code) involved using WIM register to set windows used by task A to be invalid (so their use would trigger a trap), and changing the ASID value - so if task B was already in TLB you'd avoid flushes, or reduce them only to flushing when running out of TLB slots.
The "golden" target for fast-path calls was calls that would require as little stack as possible, and for common services they might be even kept hot so they would be already in TLB.
Is that it?
What makes doors "potentially worse" than regular threads?
The kernel can spin up a thread on the server side, but this works just like common thread pool libraries, and I'm not sure the kernel has any special role here except to optimize context switching when there's no spare thread to service an incoming request and a new thread needs to be created. With a purely userspace implementation there may be some context switch bouncing unless an optimized primitive (e.g. some special futex mode, perhaps?) is available.
Other than maybe the file namespace attaching API (not sure of the exact semantics), and presuming I understand properly, I believe Doors, both functionally and the literal API, could be implemented entirely in userspace using Unix domain sockets, SCM_RIGHTS, and mmap. It just wouldn't have the context switching optimization without new kernel work. (See the switchto proposal for Linux from Google, though that was for threads in the same process.)
I'm basing all of this on the description of Doors at https://web.archive.org/web/20121022135943/https://blogs.ora... and http://www.rampant.org/doors/linux-doors.pdf
(the descendants of that proposal is `sched_ext`, so maybe it is possible to implement doors in eBPF + sched_ext?)
There isn't a door_recv(2) systemcall or equivalent.
Doors truly don't transfer messages, they transfer the thread itself. As in the thread that made a door call is now just directly executing in the address space of the callee.
They're more like i432/286/mill cpu task gates.
In somewhat anachronistic verbiage (at least in a modern software context) this may be true, but today this statement makes it sounds like code from the caller process is executing in the address space of the callee process, such that miraculously the caller code now can directly reference data in the callee. AFAICT that just isn't the case, and wouldn't even make sense--i.e. how would it know the addresses without a ton of complex reflection that's completely absent from example code? (Caller and callee don't need to have been forked from each other.) And according to the Linux implementation, the "argument" (a flat, contiguous block of data) passed from caller to callee is literally copied, either directly or by mapping in the pages. The caller even needs to provide a return buffer for the callee's returned data to be copied into (unless it's too large, then it's mapped in and the return argument vector updated to point to the newly mmap'd pages). File descriptors can also be passed, and of course that requires kernel involvement.
AFAICT, the trick here pertains to scheduling alone, both wrt to the hardware and software systems. I.e. a lighter weight interface for the hardware task gating mechanism, like you say, reliant on the synchronous semantics of this design to skip involving the system scheduler. But all the other process attributes, including address space, are switched out, perhaps in an optimized matter as mentioned elsethread but still preserving typical process isolation semantics.
If I'm wrong, please correct me with pointers to more detailed technical documentation (Or code--is this still in Illuminos?) because I'd love to dig more into it.
FWIW, Here's the Solaris man page for libdoor: https://docs.oracle.com/cd/E36784_01/html/E36873/libdoor-3li... Did you mean door_call or door_return instead of door_recv?
It's like a system call instruction that instead of jumping into the kernel, jumps into another user process. There's a complete swap out of code and data in most cases.
Just like with system calls how the kernel doesn't need a thread pool to respond to user requests applies here. The calling thread is just directly executing in the callee address space after the door_call(2).
> Did you mean door_call or door_return instead of door_recv?
I did not. I said there is no door_recv(2) systemcall. The 'server' doesn't wait for messages at all.
Later in this article authors says tat Server manage its own pool (optionally bounded) of threads to serve requests.
This means that when you do a Door IPC call, the service routine is called immediately, not at some indefinite point in time in the future when the server process gets picked by scheduler to run and finds out an event waiting for it on select/poll kind of call. If the service handler returns fast enough, it might return even before client process' scheduler timeslice ends.
The rapid changing of TLB etc. are mitigated by hardware features in CPU that permit faster switches, something that Sun had already experience with at the time from the Spring Operating System project - from which the Doors IPC in fact came to be. Spring IPC calls were often faster than normal x86 syscalls at the time (timings just on the round trip: 20us on 486DX2 for typical syscall, 11us for sparcstation Spring IPC, >100us for Mach syscall/IPC)
EDIT:
[1] Some might remember references to 1:1 and M:N threading in the past, especially in discussions about threading support in various unices, etc.
The "1:1" originally referred to relationship between "kernel" thread and userspace thread, where kernel thread didn't mean "posix like thread in kernel" and more "the scheduler entity/concept", whether it was called process, thread, or "lightweight process"
One potential problem with regular IPC I see is that it's nondeterministic in terms of performance/throughput because you can't be sure when the scheduler will decide to run the other side of whatever IPC mechanism you're using. With these "doors", you bypass scheduling altogether, you call straight "into" the server process thread. This may make a big difference for systems under load.
One can implement async doors by having the client transfer the FD to a door where the client is the service, then have the server do what it can synchronously, schedule async work, return immediately, and later call back the client on its door when the slow work is done. But if you'd do this then you might as well just use more traditional (and portable) IPC techniques.