I know people tend to hold up POSIX compliance as something that's of utmost importance, but from my conversations with Bionic's maintainers, there's an awful lot of cruft and poor designs in some of the POSIX APIs.
Lots of APIs are fundamentally unusable in the presence of threads (e.g. there's no way to modify environment variables safely). This is a bit excusable, because pthreads were bolted on a decade after everything was already ossified.
Lots of APIs are a giant pain in the ass to use compared to alternatives provided in Linux/BSD (e.g. alarm vs timerfd or kqueue).
A few APIs are massive footguns that basically shouldn't ever be used. It's impossible to use POSIX shared memory without leaking. pthread_cancel is insane.
1: http://austingroupbugs.net/view.php?id=529
2: https://www.gnu.org/software/libc/manual/html_node/Reading_0...
> jmgao
jesus, you guys come out of the wood work, huh? :D
My main takeaway from the article is POSIX is too antiquated and anemic as a "system API" for modern-day use - however I'm surprised the article doesn't seem to touch upon POSIX shortcomings instead being a consequence of the UNIX design philosophy - a strong personal concern of mine regards UNIX-like's future development course - should we continue to uphold "everything is a file", ioctl(), and piping text-streams as the best way to build an OS?
Curiously, the article chides POSIX for lacking modern abstractions (specifically, graphics) but I disagree - POSIX would be a disaster if it included anything like OpenGL or Cairo (case-in-point: Win32 is tightly coupled with GDI and now we're paying the price by being physically unable to do deep-color graphics in Win32 without serious hacks) - I think that (to paraphrase one of the UNIX philosophies) "small APIs that do one thing and do it well" applies to POSIX (as it does process and system management, and nothing else) and it has worked.
Coming from a modern Windows developer experience - I think there's a great deal to be learned from the failures of the vaporware "object-oriented operating system" concepts of the mid-1990s and how the concept has been resurrected in the form of PowerShell object-piping and WinMD today and should be explored as the next possible step in OS design at a fundamental level - not just as a shim on top as it is with Windows.
That said, UNIX and GNU/Linux have always evolved in small steps - pretty much every great-leap-forward is doomed to failure or endless ideological controversy (see `systemd`) so maybe UNIX-like OSes are not the right place to test new design concepts - but some new OS will - and eventually that OS will supersede Linux in the way many people believe Rust will replace C++.
POSIX doesn't do a very good job of upholding that. POSIX is full of entities (e.g. notably processes and threads) which can't be manipulated via file descriptors. Some POSIX implementations do better – e.g. FreeBSD's pdfork – but that isn't standardised. By contrast, Windows NT in which "everything is a handle" actually provides a more unified interface to OS objects than POSIX does.
Well there is /proc
Also, Linux /proc doesn't really allow you to treat processes as file descriptors like FreeBSD's pdfork does. Sure, I can open /proc/123, but operations on the resulting file descriptor have no relationship with the process – if I pass the FD into a select() I won't get any notifications about the process lifecycle.
The standard way to exchange (read, write) information should be through the Unix file interface where possible. Note that "file" here means "FILE *", i.e. an "opened file" or "stream". Not a file on a filesystem which is just one way to make one kind of stream.
In other words that mantra is about simplicity. Simplicity is what enables software to exist.
I don't agree. POSIX systems provide APIs to wait on events on file descriptors – select, poll, etc. I can use those APIs to wait on an event on a pipe, a network socket, a device file, etc. To wait on an event on anything which is represented by an file descriptor.
But what if I want to wait for a child process to exit? I can't use select/poll for that since processes are not represented by file descriptors in standard POSIX – FreeBSD is a notable exception, but FreeBSD's facilities are non-standard.
If I want to simultaneously wait on both a child process to terminate and a message to arrive on a socket, I am forced to use multiple threads. For example, I could have a thread to execute waitpid() and then have it write to a pipe, and have another thread select/poll on both that pipe and the network socket. If processes were represented by file descriptors, I could have a single thread doing a select/poll on both the network socket and the child process.
> In other words that mantra is about simplicity. Simplicity is what enables software to exist.
Yes, and POSIX fails to be simple here. By failing to sufficiently unify its abstractions, it forces application code to be more complex than it should be.
And Linux has signalfd (also not POSIX).
I agree with you, signals are a pain, and could probably be replaced by fds at the handling side (of course not: KILL, STOP, CONT...). The overhead of polling vs asynchronous signal delivery could be optimized away with the VDSO.
> If I want to simultaneously wait on both a child process to terminate and a message to arrive on a socket, I am forced to use multiple threads.
As a matter of fact, you are not. I'm sure you know about EINTR.
Anyways I was more referring to your original parent ('should we continue to uphold "everything is a file"'), and it seems we both agree that "everything is a file" as a pure ideology is very worthwhile.
The problem with using signals to manage child processes comes when child processes are started by libraries. If the library installs a handler for SIGCHLD, it could conflict with other libraries or with the application which wants to do the same thing. The great thing about FreeBSD style process descriptors is that no signal handlers are needed–a library can start a child process and then wait for it to complete (and wait for other things such as network sockets simultaneously) without relying on signal handlers which don't work well when an application is composed of numerous independently developed libraries (as many modern applications are).
I don't understand that ideology - I don't believe everything can be abstracted as a file considering that ioctl() breaks that abstraction. For example, what is it to "read from /proc/{pid for Chrome}"? Would you be reading from the process' memory space? Reading from a rendered bitmap of the process' main framebuffer? Reading a human-readable text file of process metadata? If it's labelled metadata then how do you deal with UX localization? Same thing when "reading from a /dev device" which isn't a storage device, what should it mean to "read from a GPU"?
The OS kernel needs to provide user space processes with access to kernel-managed resources such as files, devices, network sockets, processes, threads, etc. In order to do so, there needs to be some way of identifying these individual resources to the kernel. And then the question is, should every type of resource have a distinct type of identifier? Or should we have a single type of identifier which could refer to an instance of any one of those types of resources?
The pure "everything is a file descriptor" ideology (or its Windows NT equivalent, "everything is a handle") says we should have a single type of identifier, the file descriptor (or handle), which can represent resources of any type for which the process can invoke kernel services – processes, threads, files, network sockets, etc.
Standardised POSIX does a poor job of living up to this ideology, since APIs for managing processes (kill, waitpid, fork, etc) take and return PIDs, not file descriptors. Since a process is not a file descriptor, I can't select()/poll() on it. Using PIDs is also prone to race conditions, whereas file descriptors are less prone to this problem (although not completely immune from it.)
The process descriptor functions such as pdfork provided by FreeBSD do a much better job of living up to "everything is a file descriptor" ideology than pure standardised POSIX does.
> I don't believe everything can be abstracted as a file considering that ioctl() breaks that abstraction. For example, what is it to "read from /proc/{pid for Chrome}"?
Why must every file support read() and write()? Some device files or other special files might only support ioctl(), and maybe also select() and poll(), and I see nothing in principle is wrong with that.
The idea is rather that there should exist a file you can read the process' memory space from, which there is. The path to and specific format of that file is specified by the system documentation.
I can generally get more done faster with a binary pipe and some sort of generator for serialization than with higher level communication APIs.
COM is Windows and COM is about as purely object oriented as you can get. It's not vaporware.
A great example of how all of this turns out to be good design is that you can see how easily UNIX-based systems adopted Unicode. Say what you like about byte streams as a universal interface, but it sure was easy to switch from ASCII to UTF-8 when your system APIs weren't all built around the idea that everything would be encoded in a particular way.
I don't think systemd is unpopular or controversial because it's a great leap forward. That's just buying into the propaganda. It's unpopular and controversial because it presents itself as a great leap forward while not being one.
>but some new OS will - and eventually that OS will supersede Linux in the way many people believe Rust will replace C++.
C++ hasn't even replaced C. Why would Rust replace C++? When has a popular, stable, architecturally-important language ever been replaced?
The sort of people that think that Rust will replace C++ are the sort of people who thought Java would replace C++, who thought C++ would replace C, who thought Windows would replace Unix and who think that another system of opaque proprietary object-oriented APIs will replace POSIX. They're wrong, in every way.
All of these have happened, for many use cases.
How did you come to that conclusion? The bulk of the controversy I've seen about systemd has nothing to do with hype, the controversy I've seen has largely been focused on its invasiveness.
Its designers are also very anti-open-source, disliking any element of choice. They also seem to care only about desktops.
Nope, as I said before the controversy is about its invasiveness. Read almost any article criticising systemd and you'll find people expressing something along the lines of 'what started as an init system has grown massively into something that resembles its own layer of the OS design'.
Unfortunately the hierarchical structure is lost in the name. It's one flat string and '/' and NUL have special meaning.
In a way, containers show that there is a legitimate need for multiple roots.
You still want to restrict resources (number of cores, memory, etc).
Nevertheless, I agree that containers are often a cut through Gordian Knot. Just put everything in its own box instead of resolving dependencies.
For that matter, I think it's naive to say that containers and namespaces are the same thing.
Actually, there’s quite a bit about that.
Having actually a filesystem API that takes a UUID for a partition, and then a relative path within the partition, would solve a lot of problems.
Hell, if you want to do that, you can do it. But there's no reason to want to do that. Partitions are an implementation detail of the filesystem. You shouldn't care whether /home is on the same partition as / or not.
POSIX added openat and related calls that operate relative to a specified directory descriptor, rather than the root of the mounted directory hierarchy.
It would also be possible to take this a step further and have calls which work with a filesystem descriptor and work on a filesystem independently of the mounted filesystem hierarchy. If it was possible to mount a filesystem without binding it (getting a filesystem descriptor), and then separately bind that filesystem into the mounted filesystem hierarchy using the descriptor. This would enable the direct use of a filesytem without it being bound, and that would allow for example private use of a filesystem by a process without it being globally visible, while might be useful for transient access to storage media without any potential for races (nothing else could open files or have a CWD in the filesystem).
I'll acknowledge that this is not strictly necessary, but it is an area where the underlying design is not well abstracted and is inflexible--sophisticated use of bind mounts is difficult, and the mount(8) dance to use them is terrible, all because it has to work around the lack of separation between mounting and binding.
If your software is hardcoded to a single path, then it will install only on a single drive.
The idea is intended to allow specifying a path that will be globally unique.
Windows now runs all the workloads that Sun, SGI, DEC, HP, IBM yadda yadda workstations once ran. I don't remember the last time I saw a real Unix workstation outside my home office where I keep an Octane for nostalgia's sake, and the wife has her old SPARCstation.
But you are right about C and POSIX.
Most of us have Windows stations. We can install anything on them, but most of us didn't bother. They're glorified remote terminals anyway (the code is compiled and edited on a bunch of build servers). The servers do run Linux, but if we were to start from scratch now, we could probably pull off a decent build system under Windows as well. I'm using Emacs and relying on a bunch of Unix tools, either directly (grep) or indirectly (cscope through xcscope), but my colleagues who use Eclipse wouldn't feel much of a difference if the servers ran Windows.
I used to run Linux at my previous $workplace, but I definitely lost more time than my employer would be comfortable knowing disentangling things that regularly broke after updates (mostly Gnome- and systemd-related, really, but thanks to xdg's lovely practices, using a Linux system without something that can perform the black magic associated with sessions, seats, file associations and whatnot is very unpleasant).
My Windows laptop hasn't crashed once, the applications I need haven't broken once, and I haven't had to do the restart your computer to install this driver dance at all, I literally started it and hacked away. Plus, doing systems programming on a Windows system isn't all that bad, thanks to Dave Cutler and his colleagues being brilliant engineers.
I'm torn about using Windows at home, largely because I now have fifteen years' worth of BSD and Linux-managed files and countless convenience scripts to migrate, because I don't really trust Microsoft and because I think fighting my OS so that it will stop giving me ads and sending my stuff to Redmond is about as productive as wrestling with a modern Linux systems' innuvashon. But, much unlike 15 years ago, I probably could.
> my home office where I keep an Octane for nostalgia's sake, and the wife has her old SPARCstation.
I wish I had neighbours like you :-(
Sounds like you were upgrading too often (almost sure, if you were already dealing with systemd). Windows releases a new version only every 3-4 years, there's no reason to update your Linux workstations any faster. At my work, we're still using a mix of Ubuntu 12.04 and 14.04, both of which are still supported, and we just add a couple of repositories for specific applications (particularly browsers).
I do use Debian Unstable on my personal laptop, but that's because I don't mind fixing it if I have to (which, mind, I haven't had to do in a long time). But for work? LTS all the way.
I used to run Debian stable at home a while ago and did like the stability, but the security update situation is not exactly something I'm happy with.
As for Debian security updates, what do you mean?
I don't want to minimize or belittle the work that they're doing, I only mention Debian because it's been my go-to distro for a very long time. They're also alleviating the problem in the most common use cases (e.g. they do update Chromium if you need a webkit browser). Codebases like WebKit's are simply too large, too complex and too quickly-shifting for a community-driven project to be able to backport fixes.
Even where the codebase is small enough, backporting is a nasty business. I've seen it done commercially, so with proper funding and proper teams and whatnot, and the success rate is not something that I'd consider encouraging. I've shot myself in the foot while doing it, too.
There are certain types of setups that lend themselves well to long-term support models. Server systems, up to a certain degree of complexity, embedded systems with a restricted set of packages -- maybe. A modern Linux desktop is not one of these systems IMO. A Linux desktop with four year-old packages is very likely to be very buggy in very nasty ways.
However, the importance of POSIX isn't really to do with specific design details, but rather because it aids in software portability. This is still a desirable trait. In other words, the answer is to design a better portability solution (N.B. For anyone thinking of posting it, the XKCD standards link is not relevant here, you'd want to design something that had support from key stakeholders before it was implemented, there's no requirement to fragment the market).
Practically every non-trivial POSIX-ish program is not portable. This also includes quite some supposedly higher level applications written using things like Java, Go, Ruby, Python ... since they all leak POSIX details all over the place and make non-portable use quite easy.
How so?
Android is mostly Java, the NDK only white lists a restricted set of libraries that aren't POSIX related, Android 7 dynamic linker will kill any app that tries to link to device libraries not part of that list, the Linux fork lacks APIs like e.g. UNIX IPC.
Android Things is even tighter. Only user space drivers are allowed, with C support just added this week. Before only Java drivers were allowed.
Magenta/Fuchsia is a micro-kernel with Dart as main application stack.
Uh, libc?
> Android 7 dynamic linker will kill any app that tries to link to device libraries not part of that list
Yes, it prevents you from depending on things that aren't required to be there. Apps depending on the device's OpenSSL libssl.so were horribly broken when Android switched to BoringSSL after Heartbleed. Preventing apps from loading system libraries guarantees that they won't break when those libraries are changed or removed in future releases.
> the Linux fork lacks APIs like e.g. UNIX IPC.
POSIX/SysV IPC are unusable on any system with untrusted applications. They live in a global namespace not tied to a process, and there's no reasonable way of doing any sort of accounting on them. There isn't even a way to use them correctly. If you shm_open, and fail to shm_unlink (for example, if you're OOM killed), the shared memory object will live forever. In the specific case of shm, Android introduced a far superior API, ashmem, which tracks shared memory objects as file descriptors, instead of manually refcounting names in a global namespace (memfd was added in linux 3.17, which solves the same problem in the same way).
libc is the C runtime library, a subset of POSIX.
A conforming ANSI C implementation is not required to provide headers like unistd, or any kind of POSIX compatibility beyond what ISO/IEC 9899 requires.
> POSIX/SysV IPC are unusable on any system with untrusted applications.....
Either POSIX compatibility matters, or it doesn't.
> A conforming ANSI C implementation is not required to provide headers like unistd, or any kind of POSIX compatibility beyond what ISO/IEC 9899 requires.
Yes, and Android's libc provides almost all of (all of? I don't know for sure) POSIX.1.
> Either POSIX compatibility matters, or it doesn't.
POSIX IPC is a non-mandatory option in POSIX. See http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_... if you want to be a language lawyer.
More to the point, POSIX compatibility matters up to the point where it's useful. Implementing the shared memory functions by returning -1 and setting ENFILE is a perfectly compliant implementation. It's also worse than just not providing an implementation.