Time to get the Posix elephant off our necks?

"The Socket API, local IPC, and shared memory pretty much assume two programs"

There's no truth to this whatsoever - it's an utterly absurd statement because:

* The socket API as applied to networking is empirically designed for arbitrary numbers of peers -- there are unlimited real world examples. The socket API as applied to local IPC is of course similarly capable.

* Shared memory - mmap() is about as generic of a memory sharing interface as one could hope for. It certainly works just fine with arbitrary numbers of clients. There are again countless examples of using shared memory with arbitrary participants -- for example all the various local file databases: sqlite, leveldb, berkeley db, etc.

"We should be able to assume that the data we want exists in main memory without having to keep telling the system to load more of it."

Yes, this is exactly what mmap() already does. mmap() is part of POSIX. Use it!

"Kode Vicious, known to mere mortals as George V. Neville-Neil,"

Oh, ok.

First, memory models. The author seems to be arguing for some way to talk about data independent of where it's stored. The general idea is that data is addressed not with some big integer address, but with something that looks more like a pathname. That's been tried, from Burroughs systems to LISP machines to the IBM System 38, but never really caught on.

All of those systems date from the era when disks were orders of magnitude slower than main memory, and loading, or page faulting, took milliseconds. Now that there are non-volatile devices maybe 10x slower than main memory, architectures like that may be worth looking at again. Intel tried with their Optane products, which were discontinued last year. It can certainly be done, but it does not currently sell.

The elephant in the room on this is not POSIX. It's C. C assumes that all data is represented by a unique integer, a "pointer". Trying to use C with a machine that does not support a flat memory model is all uphill.

Second, interprocess communication. Now, this is a Unix/Linux/Posix problem. Unix started out with almost no interprocess communication other than pipes, and has improved only slightly since. System V type IPC came and went. QNX type interprocess calls came and went. Mach type interprocess calls came and went. Now we have Android-type shared memory support.

Multiple threads on multiple CPUS in the same address space work fine, if you work in a language that supports it well. Go and Rust do; most other popular languages are terrible at it. They're either unsafe, or slow at locking, or both.

Partially shared memory multiprocessor are quite buildable but tough to program. The PS3's Cell worked that way. That was so hard to program that their games were a year late. The PS4 went back to a vanilla architecture. Some supercomputers use partially shared memory, but I'm not familiar with that space.

So those are the two big problems. So far, nobody has come up with a solution to them good enough to displace vanilla flat shared memory. Both require drastically different software, so there has to be a big improvement. We might see that from the machine learning community, which runs relatively simple code on huge amounts of data. But what they need looks more like a GPU-type engine with huge numbers of specialized compute units.

Related to this is the DLL problem. Originally, DLLs were just a way of storing shared code. But they turned into a kind of big object, with an API and state of their own. DLLs often ought to be in a different protection domain than the caller, but they rarely are. 32-bit x86 machines had hardware support, "call gates", for that sort of thing, but it was rarely used. Call gates and rings of protection have mostly died out.

That's sort of where we are in architecture. The only mainstream thing that's come along since big flat memory machines is the GPU.

Another annoyance is that passing data between C and another language is hard. For instance, if you want to pass a Swift string in to C, you need to be careful that Swift doesn't free the string while C is using it. The "solution" to this is to have explicit methods in Swift that take a closure which guarantee the data stays alive for the duration of that closure. On the C side, you need to copy the data so that Swift can free the string if you need to keep it longer than that one block. Going from C to Swift is also a pain.

A cool thought is: what if the OS provided better memory management? What if it had a type of higher level primitive so that memory could be retained across languages? For instance, if I pass a string from C to Go, why do I need to copy it on the Go side? Why can I not ask the OS to retain the memory for me? Perhaps we need retain / release instead of malloc and free. Anyway, just a random thought.

[1] https://www.amazon.com/Design-Implementation-FreeBSD-Operati...

reading from stdin isn’t challenging, nor writing to stdout. if someone can’t imagine why that might be useful, then i’d argue their journey as a software engineer is either at its end, or right at its beginning.

at the cost of potentially sounding inflammatory, “get good”.

Programming is basically always just a matter of loading data, transforming data, and then putting it somewhere. The simplest record keeping systems do that, and the fanciest search algorithms do that. Decode/encode/repeat.

EDIT: The beauty of UNIX to me is the interoperability of the text stream. Small components working together. Darwinesque survival of the fittest command.

And not even the obvious faults, pointed out by others here. There's the question of an apparent complete ignorance of OS research, for example systems that rely on h/w memory protection and so can put all tasks (and threads) into a single address space. But what's the actual take home from such research? The take home is that these ideas have, generally speaking, not succeeded, and that to whatever extent they do get adopted, it is incremental and often very partial. If you don't understand why most computing devices today do not run kernels or applications that look anything like the design dreams of 1990-2010 (to pick an arbitrary period, but a useful one), then you really don't understand enough about computers to even write a useless article like this one.

Most languages have a way to slurp a file into memory in a single function call, after all. The fact that files exist shouldn't be a barrier here.

https://wiki.lesswrong.com/wiki/Chesterton%27s_Fence

Software is so huge that it would take you a lifetime of programming from different perspectives to get a grip on what it really is. So we are all doomed to experience POSIX through whatever programming experience we end up getting deep in.

I feel the underlying problem most software is that it's just too damn complex, so you can't fit enough of it in your head to design it how you think it should go. An average person can't go "Oh, I think the kernel should be able to do this" and then go whip it up and having an experiment running in a little bit. That's an esoteric corner full of tons of specialized and arcane knowledge that, truth be told, is completely invented. And half of that invention is workarounds for other bad inventions.

I dunno enough to just pronounce doom on POSIX, but I do feel like the rickety C way of doing things (everything centered around intricately-compiled machine code executables, incredibly dainty fragile eggshells that shatter and spill their entire guts of complexity on the world) underpins a ton of the problem.

The number of years you would need to just read, let alone grok all the hundreds of millions of lines of code that run on our system is just beyond human lifetimes now.

I read a call for innovation, the general thrust of which is based on the (obvious?) argument that if you build every new system to be compatible with the old one you limit the capabilities of the new.

Of course, the customer requesting that compatibility gets what they want -- an easier time building/porting software.

Alternate reading: ~POSIX (or just having a standard) is so useful that everyone wants it everywhere all of the time and I want something better.

- there could be better primitives for communication than a byte stream. Why can't I just send another process a chunk of memory?

- the posix byte stream API is bizarrely based on a flat buffer instead of a ring buffer, which is just the obviously wrong data structure IMO.

- Too much of the threading API is preempted threads, when cooperative scheduling (aka yielding) is so much easier to implement and understand.

The author doesn't really bring up concrete alternatives like the above though, so it's hard to know what they are railing against

Based on the other responses here, it sounds like the author should keep in mind that you can't get your elephant off your neck and eat it too, but maybe I shouldn't open that can of worms lest I attract some early birds.

Isn't this what Intel was trying to do with Optane? What happened with that? It seems like a great idea.

For example, the part about how it would be nice to assume data exists in main memory, instead of accessing it through the filesystem? That's existed for nearly 40 years, it's called mmap, and every modern operating system implements it. You can trivially write a wrapper that automatically mmaps opened files if you want, but creating a MAP_FILE memory mapping is one of the easiest things you can do in C.

Likewise the stuff about how IPC, sockets, and shared memory assume that a resource is only going to be shared between two programs. The whole statement is weird because all of those things can be used by more than two programs, and it's not clear what the author means by saying they're designed for a single receiver/sender. The Berkeley sockets API is admittedly a bit intimidating at first but that's because it covers a lot of functionality that people actually need. It can handle UDP sockets, TCP sockets, but also other protocols and more exotic things like netlink sockets, sctp sockets, and can be extended to things like John Ousterhout's proposal about replacing TCP, as evidenced by the fact that John Ousterhout actually implemented Homa on Linux. Using shared memory on Unix isn't exactly simple but that's because it's doing something complicated. When multiple processes want to share memory there needs to be some coordination to establish what needs to be mapped, to grant privileges, etc. The SysV shared memory APIs suck but on Linux things are much better now with the memfd system calls. And the vast majority of people who need things like sockets or shared memory are going to be writing code in a higher level language anyway, using libraries that abstract the low level details. The article implies that it's a problem that people need to use these abstractions in the first place, but it seems asinine to me when there isn't an actual proposal of how things would be better, and the hand waving completely ignores things like the fact that realistically low-level C APIs are needed to have any hope of writing interfaces that can be accessed by multiple languages.

Furthermore the design of Unix allows new system calls to be added. If you have an idea for something that is way better than sockets you're free to implement it and add new system calls for it. This is exactly why there are many ways to do the same thing on Linux (e.g. establish shared memory regions), because people come up with new and better ideas and the ones that are actually performant, useful, and secure are the ones that get merged and added to Linux. Maybe there really are some brilliant ideas for new ways of doing things that need a ground-up redesign and can't be bolted onto Unix, but if that's the case people should be able to clearly elucidate what those ideas are and WHY they won't work with Unix.

One final thing I'll say about Unix is that for all its warts, it's very fast. This is a point that's been made a million times before, e.g. in "Worse Is Better" and "The UNIX Hater's Handbook". There have been lots of interesting alternatives to Unix developed, but the reason none of them have taken traction is that at the end of the day no one wants to use something that is newer, less well known, less well tested, and significantly slower. Plan9 and Fuchsia are cool, but they are much slower than Linux and it's not clear if it's really possible to fix them. Users want their computer to run quickly, they want their phone to run quickly, and they want good battery life. Big companies want their applications to run quickly and spend as little on hardware as possible. All of these things mean that pretty much any Unix alternative is a non-starter unless it is at least close to the performance of existing Unix implementations, including Linux and macOS I suppose. A big explanation of the success of Unix has been that it has co-evolved alongside modern hardware. It's hard to do something that is radically different and takes advantage of modern hardware when you're writing a general purpose operating system (there are absolutely exceptions though for operating systems serving niche use cases).

It's one thing to complain when there are no choices. It's quite another when there seem to be genuinely good alternatives in the space (Apache Arrow is not even the only option).

I think there are significant advantages that will accrue by moving toward systems designs that encourage "more zero copy plumbing" -- not totally clear on all the ways the posix legacy might prevent that but the parts of posix that do prevent that will prove to be the most important pieces to shed I think.

The main gist is: Why is memory shared between two processes?

The answer is: you can unblock a deadlock when you have a consumer and producer by just turning one of them on and off. In all other cases you can't. All the 'solutions' to that problem make assumptions about the processes sharing state which are somewhere between difficult to impossible to keep valid in the real world.

In short: 2 is a magic number and it's not the fault of posix.

[1] https://github.com/Zeal8bit/Zeal-8-bit-OS#user-space

That said modern OSs do provide additional APIs that are higher level, but in general they are all just opaque object wrappers around the same core principles, just less tethered to the limitations of 80s era C.

May be provide “addr = mmap_open(path, flags, &size)” ?

Given POSIX is a system programming interface, aimed at no specific use case whatsoever, maybe the issue is that they are using this directly rather than going via some sort of higher level abstraction, a library framework which would provide them with the capabilities that they need?

Exposing system APIs like ErrorOr (like the Rust Result or the Maybe monad) resolves an entire class of bugs (take fork/kill/-1 as an example [1]) as well as ambiguity like "do I check the return value for the reason, or do I check the magical errno variable?". Dropping general OS compatibility also allows for other changes, like object oriented interaction with the GUI system and other system libraries instead of relying on having the namespacing happen in the name of a function call.

I know SerenityOS won't ever replace any operating system we use today, but it's a nice demonstration of the API features we could all be using.

Sometimes I wish Windows' APIs were more POSIX-like, but Win32 has some pretty useful functions built right into the OS that you'd need to introduce DLL/dependency hell/static compilation for if you're trying to stick to POSIX only APIs. Think common concepts like "the clipboard" or "the current resolution of a display" without needing to link to a library or access a file like /dev/fb0 (is there a /dev/fb1? when do you use it?).

Mobile platforms (Android, iOS) with special lifecycles also allow for things like "the app resumes where you left off" and "the app suspends to disk when the system is under load" natively without requiring developers to build their own save/restore state mechanisms. Imagine having to build a system like that on your average Linux desktop with mere POSIX APIs, you'd go crazy with the complexity required.

Perhaps what programs runnong on Unix-based operating systems really need is a wrapper around POSIX and its low-level implications. GTK and Qt provide many such APIs for free, for example, including stuff like networking and other I/O. GTK comes with its annoying particularities (CSD, and all the other GNOME decisions) and Qt licenses are either VERY free or VERY expensive, making them incompatible with other projects.

With the Linux space transitioning from SystemV+ALSA/Pulse+X11 to systemd+PipeWire+Wayland, perhaps the space is ripe for a new, more modern wrapper library for native languages.

[0]: https://github.com/SerenityOS/serenity

[1]: When fork() fails, the call returns a negative number. When you try to kill() a PID of -1, you kill every process your current UID has the permission to kill. Programs failing to fork, shutting down, and killing what they thought was a fork()ed process can accidentally end up killing all the open applications for a user.

2. ???

3. We should design new APIs with abstractions that do not represent either modern nor PDO architecture.

I feel like the author should spend more time on 2 to make his case.

Also, the actual answer to the question posed is: you might not think data management is an important part of your job, but it’s actually more important than the analysis, so you spend more time on it.

This already exists in POSIX. It's called mmap, and approximately no one uses it. Maybe the article should have explored why, and suggested how to solve that problem, instead of insisting we must throw everything away and this will magically solve our problems.

The actual problem of the described organization is there data organization. Not UNIX.