Towards userspaceification of POSIX – part I: signal handling and IO

Yeah per thread virtual memory is called a process. Within a process the memory manager has to maintain a single virtual address space. You could do it with a red black tree and a global lock. Or you could use potentially cleverer algorithms. Skip lists are friendly to scalable concurrency. You could also do a radix tree of page tables, similar to what x86-64 processors do internally, then have each item be its own atomic long, and use some kind of optimistic transactional locking strategy when allocating mappings that span multiple pages. Personally, I think the rbtree is easiest, since it's better to leave memory scalability to malloc(). The way I like to make malloc() scale is having an arena for each CPU and then indexing them using sched_getcpu().

> To have a different sub-tree, you must have a different root. However, you can share sub-trees amongst different trees.

That's what I meant by each thread having their own map (i.e. root), and within each map they have a shared branch, a branch being a pointer to a subtree. In this case, what I had in mind was the shared branch being at the root; basically slicing the user land address space in half (or 1/radix size...), though, that's probably getting too detailed.

Perhaps by "branch" you thought I meant a jump operation in code. That's certainly not how most MMUs work, which walk a data structure constructed by the OS. But some MMUs actually fault to OS provided code to resolve an address, in which case "branch" in the meaning of software code isn't necessarily wrong. (But to be clear, it's not what I had in mind.)

There are a few existing libraries like libancillary that would do this for you and provide some level of OS compatibility.

> the whole BSD socket API is plain horrible; it only takes a couple of uses to realize that you never want to use it directly again

That was my initial impression as well, but recently I’ve had to use it again and surprisingly did not find as bad as I remembered. Except, indeed, for the fd-passing experience, for which see my wrapper elsewhere in the thread (also other sideband stuff, but how often do you really need SCM_CREDENTIALS?).

The syscall/kernel-ABI people seem to love it as well—I remember reading an article that praised it for remaining so stable over its lifetime. I think these are actually two sides of the same coin: BSD sockets essentially layer a second ABI on top of C function invocations. It’s a tad more specific than generic ioctl-ish (selector, payload), but not that much, and the farther away you are from the happy path of send()/recv(), the closer it is to that (and the more extension capability the kernel programmer wants, and the more misery the userland programmer feels).

The Unix approach of exposing syscalls from libc essentially directly was a nice thought, but the sockets API feels like a reductio ad absurdum of it.

Rust does however implement the IO traits for `&File` as well (shared), and IIRC also implements `try_clone` which is the dup equivalent.

One of the reasons the FD interface is so complicated is because there are many of the same operations, but they do different things to different underlying kernel implementations. In Linux, you have no standard way to tell what the underlying FD is. It's such a wide surface, different kernel surfaces might implement some of the many different file APIs (polling especially) slightly differently. In many cases, you can reuse standard tooling, but random ioctl calls mean you can't always reuse tooling. Nonstandard implementations of standard file calls can make it dangerous if you don't know which type of FD you have. So the good thing is they are standard, you use the same set of tools (file APIs) to operate with them, so it makes the system interface smaller and simpler, but it lacks the fidelity to create orthogonal meta programming over them. It sounds like an unimportant complaint that you can't tell the underlying type, but it also means most languages refer to FDs indiscriminately. You don't typically get type safety in languages for FDs (e.g. different class for an FD backed by shared memory instead of a real exclusive file), and even if you do have these classes you can't really guarantee they are correct at runtime if you get an FD from another process (e.g. over UDS from a child process), so languages leave FDs untyped.

About the global cursor, it's not an issue because you can dupe it to get a copy with a different cursor. So only references to the FD with that specific FD identifier have a location associated with itself, and that's because the OS can only look up the state on that FD. So dup gives cursor, but if an FD represents a physical seek, it might be the case that a dup'd FD actually does affect the functionality of the first FD by linking the two together, as that's a choice you can have in the kernel when implementing an FD. Dup also mean no new cursor.

So that's what I mean, the complexity of the real world means these objects are fundamentally different, and having the same APIs shoe horns them into something they're not. I've worked in large systems where you have very specific type-defined capabilities and messaging, and you end with different engineers creating many types which are equivalent but have independent implementations. This is another kind of nightmare because you need to convert many types to use them, which ends up being the source of a lot of boilerplate, and many things are nice, but it takes a very influential and powerful architect to curb complexity in such a system, and in the end complexity is inevitable, even if specific bouts of it spring up and get fixed, you end up with many many many APIs. So despite FDs being a very corset type API, it's brazen simplicity has eliminated an entire layer of complexity from our software.

Perhaps my complaints of FD's complexity are the misplaced pangs of an idealist, forgetting the importance of the big picture. FDs are the APIs we need, but not the ones we deserve. I think their role in the structure of our programs is far more important than their exact nature. Perhaps one day they can be replaced, but whatever does replace it will certainly have learned a lot from the humble FD.

Ah okay, good to know. I never needed to read the same file concurrently with different cursors, so that might be why the API seemed deceptively simple to me!

I have read Redox documentation; mine is in many ways very different (both the working of high-level code and of low-level code, although some goals and features are similar, e.g. "forcing all functionality to occur via file descriptors" is mandatory for all I/O in my system (although they are "capabilities"), but mine does not use namespaces like Redox does, and requires capabilities that the program uses to be passed to it in the initial message that it receives (which is the only way to receive a message without already having a capability; if the initial message does not include any capabilities then the program would be immediately terminated since it cannot do any I/O)) from Redox and from other operating systems. (Mine is also meant to be a specification independently from the implementation; so it would be possible to write an implementation in C or in Ada or whatever else, and the different components implemented independently can be used together.)

> File descriptors and capabilities are very similar

Yes, and it is what I thought too, although my idea of these "proxy capabilities" is simplified compared with UNIX file descriptors in many ways (it is something like only having UNIX sockets, created using the socketpair function, and with SCM_RIGHTS messages as well as bytes; however, there are some significant differences too).

> We're eventually going to switch fully to the openat* family of path calls

I also thought that a POSIX interface can use openat and this is better than using open etc, although my idea of operating system design does not have that either; there are no file paths.

> But that the cursor to be stored ... I'd like this state to be stored in userspace ...

Yes, and I had also considered such things. One disadvantage of requiring the client to specify the file position is that it cannot be used with non-seekable files. However, it may be possible to have a separate POSIX and non-POSIX interface (and then the POSIX interface can be implemented in the POSIX library, which will not be needed by non-POSIX programs); the non-POSIX interface might not need to use the same interface for seekable vs streaming files (a proxy can be created (in user space) if it is necessary to use a seekable file where a streaming file is expected).

I had made my own ideas independently, although there are many similarities (as well as many differences; I am describing some of them below). One difference with mine is to make a specification, and that multiple independent implementations of the specification may be made.

My ideas are like a actor model in some ways, though.

Looking at [0], my idea is very similar than Zircon kernel services. However, there are many differences than what is described by [1]. In mine, an implementation might include some additional features in the kernel, although this is just an implementation detail; user programs don't know the difference of if they are kernel services or external programs (since this is how the security model of my system is designed to work).

Mine has only one type of object for IPC rather than five; it is similar than what is called a "Channel" in Fuchsia (although it is not exactly the same, but it is a similar idea). It is the only kind of kernel objects that user processes are able to see.

In mine, process management and hardware support services are not directly exposed by the system call interface; they are only exposed by IPC channels.

Like Fuchsia, mine has no ambient authority. However, proxy capabilities are used to provide security and many other features (e.g. logging accesses, simulating error conditions, transparent compression, network transparency, revocable capabilities, etc). A program receives an initial message when it starts, and this initial message will contain IPC channels (possibly in addition to other data).

Mine does not inherently have namespaces. Files can only be accessed by capabilities, and files can contain links to other files; there are no directory structures and no file names. There is a "root filesystem" but that is only needed for purpose of initializing the system; most processes cannot see it and have no way of identifying it even if it could see it. However, when running POSIX programs, a POSIX-like namespace can be emulated by using a file containing key/value lists which work similar than Fuchsia namespaces in some ways (although such features are implemented entirely in user-mode libraries; the kernel knows nothing about them).

Also, mine does not use Unicode in any way. It also does not use JSON, XML, HTML, etc. Binary file formats are preferred; nearly everything will use binary formats. There are also many other significant differences (including UI stuff). I also consider that some other things are also no good, e.g. USB, UEFI, WWW, etc (this does not mean that it is not possible to write drivers/programs that can use them; it means that the fundamental specifications of the system deliberately avoid them, and that hardware/software designed deliberately for this system are designed to not need them).

I also would have locks and transactions, including the possibility that locks/transactions may involve multiple objects at once; this includes files, but may also need to include remote objects in some cases.

There is still i18n, l10n, a11y, etc, as well as many additional features such as "window indicators", "Common Data Format", "Command, Automation, and Query Language", etc. (The i18n does not work like Fuchsia though. For example, although language identifiers are still needed (although they are not limited to the ones included in Unicode, since it does not use Unicode), identifiers are not needed for date/time, etc (the library that deals with date/time formats can be modified to add whatever kind of calendars you want to do; the application program does not usually need the identifier for it, unless perhaps you want to reference entire months or years, but to do that requires specifying them with the data being processed by the program and is entirely separate from i18n preferences anyways).)

[0] https://fuchsia.dev/static/fuchsia-src/get-started/images/in...

[1] https://fuchsia.dev/fuchsia-src/get-started/sdk/learn/intro/...

From the point of view of the application, yes, you need to coordinate. But that’s a concern for the application. From the point of view of the kernel / device server / however your microkernel OS works, requests for disk I/O arrive in some order, probably update some cache pages or whatnot, then go onto disk’s queue. That’s arguably global, but it feels like a logical extension of the fact that you only have a single physical disk. However you’re going to multiplex it, something like that is still going to happen, and it has little to do with Unix.

What is not a natural extension of the whole thing is that, even on an otherwise completely quiescent system,

  int fd1 = open("foo", O_RDWR), fd2 = dup(fd1);

behaves very differently from

  int fd1 = open("foo", O_RDWR), fd2 = open("foo", O_RDWR);

(imagine these fds are then passed out to other processes or whatnot).

Even with O_RDONLY, these are still not at all the same. Witness the epitome of CLI design that is OpenSSL /s :

   {
          openssl x509 -out "/etc/swanctl/x509/$1.pem"
          while openssl x509 -out "$t" 2>/dev/null; do
                  fp=$(openssl x509 -in "$t" -noout -md5 -fingerprint |
                       sed 's/.*=//; s/://g')
                  mv -f -- "$t" /etc/swanctl/x509ca/$fp.pem
          done
  } < "/etc/ssl/uacme/$1/cert.pem"

This is how you pick apart a PEM cert bundle using OpenSSL: the shell spawns openssl, which reads a single PEM block from stdin, does its dirty deeds with it, and leaves stdin pointing to the next one, ready to be consumed by the next instance of itself that’s yet to be spawned by the shell. You can’t do that with a per-process file position, trivially or otherwise.

Returning to the application side, imagine you’re making a concurrent B-tree. If one thread wants to write out page X and the other page Y, they have presumably already used some locking to make sure that’ll leave the data structure consistent, so they’re free to just issue their pwrite()s and let them happen in whatever order. On the other hand, if all they have is write() and seek(), they have to hit a global lock even if X and Y are completely unrelated.

Yes, and yes.

You are not disagreeing with me.

I don't think it's possible at all. Entertain the thought. If not for anything for social reasons. A hard fork of the entire ecosystem is the only way. Every single piece of software we write today depends on every other. It's not a tree like commonly thought, it's a cyclic graph.