Hacker News Reader

Comment by throwaway09223 | Hacker News Reader

This article is just confused and wrong. Some examples:

"The Socket API, local IPC, and shared memory pretty much assume two programs"

There's no truth to this whatsoever - it's an utterly absurd statement because:

* The socket API as applied to networking is empirically designed for arbitrary numbers of peers -- there are unlimited real world examples. The socket API as applied to local IPC is of course similarly capable.

* Shared memory - mmap() is about as generic of a memory sharing interface as one could hope for. It certainly works just fine with arbitrary numbers of clients. There are again countless examples of using shared memory with arbitrary participants -- for example all the various local file databases: sqlite, leveldb, berkeley db, etc.

"We should be able to assume that the data we want exists in main memory without having to keep telling the system to load more of it."

Yes, this is exactly what mmap() already does. mmap() is part of POSIX. Use it!

"Kode Vicious, known to mere mortals as George V. Neville-Neil,"

Oh, ok.

View on HN

> * The socket API as applied to networking is empirically designed for arbitrary numbers of peers -- there are unlimited real world examples. The socket API as applied to local IPC is of course similarly capable.

I think "the [BSD] socket API" was designed for a thousand or so peers (FD_SETSIZE). That's why the API evolved. Now (2023) there are a lot of APIs and different choices you can make (blocking, polling readiness, sigio, aio, iocp, iouring), none of which is best at everything, but the non-POSIX apis are much faster than the POSIX api -- especially the ones that are harder to mix with the POSIX api.

> "We should be able to assume that the data we want exists in main memory without having to keep telling the system to load more of it."

> Yes, this is exactly what mmap() already does. mmap() is part of POSIX. Use it!

This is not what mmap() does, but the opposite. mmap() sets up page-tables for demand-paging. Those little page-faults actually trigger a check (demand) to see if the page is "in memory" (this is what Linux calls the "block cache"), an update to the page-tables to point to where it is and returns (or kick off a read IO operation to the backing store). These page-faults add up. What the author is looking for in Linux is mremap() and Darwin is mach_vm_remap() and definitely not in POSIX.

> * Shared memory - mmap() is about as generic of a memory sharing interface as one could hope for. It certainly works just fine with arbitrary numbers of clients. There are again countless examples of using shared memory with arbitrary participants -- for example all the various local file databases: sqlite, leveldb, berkeley db, etc.

But none of those "local file databases" can handle hundreds of thousands of clients, and aren't great even for the low hundreds(!), that's why most "big" database vendors avoid "just" using mmap(), and instead have to perform complex contortions involving things like mach_vm_remap/mremap/O_DIRECT/sigaction+SIGSEGV. userfaultfd and memfd are other examples of recent evolutions in these APIs.

You are looking for truth? Evolving APIs are evidence that people are unhappy with these interfaces, and these newer APIs are better at some things than the old, demonstrating that the old APIs are not ideal.

So we have evidence our (programming) model is not ideal, are we to be like Copernicus and look for a better model (better APIs)? Or are we to emulate Tolosani and Ingoli?

> You are looking for truth? Evolving APIs are evidence that people are unhappy with these interfaces,

That tracks, but this does not necessarily follow:

> demonstrating that the old APIs are not ideal.

I've been around the tech industry long enough to see several cycles of practitioners going back and forth between technologies and approaches. E.g. strong type systems (C++, Java) -> loose type systems (Python, JS) -> strong type systems (Rust, TypeScript). And each time the tide shifts there are always plenty of arguments trying to show why the previous approach was objectively worse and the new one is better.

That's not to say that nothing is moving forward, but the fact that people are unhappy with a technology doesn't mean that technology is objectively inferior to a proposed replacement. Sometimes it's just fashion.

> That's not to say that nothing is moving forward, but the fact that people are unhappy with a technology doesn't mean that technology is objectively inferior to a proposed replacement. Sometimes it's just fashion.

I apologise for writing so much that you couldn't read all of it, but this part, whilst buried at the end of the first paragraph was really important:

> > the non-POSIX apis are much faster than the POSIX api -- especially the ones that are harder to mix with the POSIX api.

That they are better in this objective way is why this particular kind of unhappiness demonstrates what I say, even if you're not otherwise interested in people being happy.

Of course we should be like Copernicus.

So POSIX as a technology may be lacking but honestly what’s impressive about it to me is that it’s also a standard, not perfect there either, of course.

I think any conversation about replacing POSIX must include an effort to update or replace the standard. Deprecate the parts that no longer work well and add things that other systems have de-facto agreed upon, at least.

" think "the [BSD] socket API" was designed for a thousand or so peers "

I think a more accurate framing is that in the 80s POSIX was designed for a thousand or so peers, and later versions have expanded and scaled accordingly. Literally every interface has these types of growth patterns so this shouldn't be a surprise.

But your point, which is the same point I made below, is that POSIX is but one of several interfaces on a modern system: https://news.ycombinator.com/item?id=34905405

As I noted below, we have semi-standardized extensions from POSIX. We don't really need POSIX to codify a new event system because we have libev/libevent. This is another point the parent article gets wrong when it says we need to get POSIX "off our necks." Nowhere does POSIX compliance stand in the way of using libev, or even using mremap(). I don't agree with your conclusion around mremap() above but it doesn't matter because the salient point is that mremap() works absolutely fine within the paradigm offered by mmap() and POSIX. libev works just fine within the paradigm offered by the file/socket api.

"But none of those "local file databases" can handle hundreds of thousands of clients"

This is absolutely not true. Those systems work just fine with even a million reader threads.

Maybe you mean the databases don't perform well under parallel writes, but this has nothing to do with the shared memory interface (how could it?). It's purely due to the design and intent of each system as primarily read-focused systems. mmap will work just as well as any other shared memory interface when it comes to reading and writing. Pick nearly any write-performant database - it's using mmap.

"Evolving APIs are evidence that people are unhappy with these interfaces"

Not quite. The actual truth is that POSIX plays very well with extensions to its interfaces and this is yet another dimension in which the above article is demonstrated to be very, very silly.

> This is absolutely not true. Those systems work just fine with even a million reader threads.

I suppose it depends how you define a "client". If you define it as a query (or an HTTP request or something like that), then I suppose you're right, but you're also not mmap()ing all the time, which is what we're actually talking about here. But if you define it the way databases traditionally do, it is itself a full-featured application, so 100 clients doing 1m queries a second is 100m queries per second, and no, mmap() isn't fast enough to do that, and since we're talking about mmap() I didn't think I needed to explain that.

> But your point, which is the same point I made below, is that POSIX is but one of several interfaces on a modern system: https://news.ycombinator.com/item?id=34905405

I must be reading a different comment, because I don't understand exactly what that has to do with anything, but I think if you're not reading "every word" of the article, maybe I'm thinking you're not reading "every word" my comments either. I was only under the impression you thought the sockets API and mmap() were ideal interfaces.

What position do you think I have that you think I need to change?

> Not quite. The actual truth is that POSIX plays very well with extensions to its interfaces and this is yet another dimension in which the above article is demonstrated to be very, very silly.

I don't understand. That POSIX is so big is one of the first complaints in the article given by implementors of embedded systems, and the "extension-like" nature of POSIX works directly against that. Then, once you have this nice "extension" like iouring with its provided buffers and files, which is basically like a whole io-state-machine operating system in zero system calls, and you know, maybe it can be part of the POSIX standardization process and make POSIX better, or maybe it just becomes popular enough there's a kevent ring extension for the bsds and there's a libiocp4 that unifies the differences, and you think that's all basically POSIX anyway, so what's the point?

Well, the point is that shouldn't stop people from imagining a system that didn't do anything else -- how much smaller the code would be, and cheaper the hardware (less ram, less rom), and maybe if you can organise the kernel with this model in mind, how much faster it would be. These sorts of ideas you can really only explore once you've mentally prepared yourself for there not being a POSIX.

"I suppose it depends how you define a "client"."

The article defines client as another thread or process - the argument is that these interfaces only support two processes and not many. There isn't any functional limit on how many processes can map the same region of memory.

"but you're also not mmap()ing all the time, which is what we're actually talking about here"

No, I don't think that's at all what we're talking about. We're talking about the ability of the architecture to scale -- not the efficiency of calling one specific syscall.

"I was only under the impression you thought the sockets API and mmap() were ideal interfaces."

I haven't said that they're ideal, only that they're demonstrably capable of enabling more than just two processes to interact.

"and you think that's all basically POSIX anyway, so what's the point?"

The point is that the article's conclusion that POSIX is preventing progress is nonsense.

As you say, standards emerge.

"how much smaller the code would be, and cheaper the hardware (less ram, less rom), and maybe if you can organise the kernel with this model in mind, how much faster it would be."

These aren't points raised by the article -- in fact so far we've been discussing the opposite (extensions and features, not cutting features).

Could things be cut from POSIX? Sure. But is implementing stuff like select() really a "yoke?" It's not exactly difficult, and I don't think you or anyone has articulated how it might hamper the inclusion of more advanced interfaces. The resources needed to implement POSIX are not significant.

What specifically do you think POSIX's architecture is preventing us from implementing?

> The article defines client as another thread or process

It does no such thing. The word "client" is given once in the text, and little is said except it isn't a thread due to the fact their support is fraught with peril and often poorly provided for by programming languages

> I haven't said that they're ideal, only that they're demonstrably capable of enabling more than just two processes to interact.

So was carrier pigeon, but sometimes we want to send messages faster than that.

> These aren't points raised by the article -- in fact so far we've been discussing the opposite (extensions and features, not cutting features).

>... about the ability of the architecture to scale -- not the efficiency of calling one specific syscall.

> What specifically do you think POSIX's architecture is preventing us from implementing?

Erm, they are the points raised by the article, that's kind-of why I think you still haven't read it yet. Here's the paragraph right after the one where it defines a client as not-a-thread (and after talking about one specific syscall):

The death of any of these systems is when a user asks, "What about Posix? How can I port my XXX program to run on your system?" Providing Posix-like semantics in these systems is their death knell, because the Posix way of thinking is so narrow and providing its varied illusions requires so many hidden changes and adherences to age-old assumptions that there is no way to have a flexible innovative system and serve the Posix elephant. This means that no matter how innovative and clever a system is, once it is tainted by Posix support, it becomes just another Posix system.

> But is implementing stuff like select() really a "yoke?" It's not exactly difficult, and I don't think you or anyone has articulated how it might hamper the inclusion of more advanced interfaces

Could you show me? I don't think it's possible to implement a select() that works with Linux's fancy new iouring that uses the provided files and buffers features easily, but I'd like to be proven wrong.

"It does no such thing."

It does. It says quite clearly, quote "If you look at most inter-process communication mechanisms, you can see that they are most often used by only two programs — usually a client and server." Two processes on a local machine, using shared memory or other IPC. IPC, not network calls.

"So was carrier pigeon, but sometimes we want to send messages faster than that."

Don't be disingenuous. You haven't articulated a faster mechanism and in fact there isn't one.

"Erm, they are the points raised by the article,"

No, they are not. Based on your replies it seems you may fundamentally not understand the points raised by the article.

"Could you show me? I don't think it's possible to implement a select() that works with Linux's fancy new iouring that uses the provided files and buffers features easily, but I'd like to be proven wrong."

You've already provided an example. The Linux kernel provides both select, a posix interface, and io_uring. Nothing about providing posix interfaces precludes providing specialized interfaces as well.

> It says quite clearly, quote "If you look at most inter-process communication mechanisms, you can see that they are most often used by only two programs — usually a client and server." Two processes on a local machine, using shared memory or other IPC. IPC, not network calls.

Two processes! Not two threads!

> Don't be disingenuous.

How would you know? You still haven't read the article, so you're arguing with a straw-man!

> You haven't articulated a faster mechanism and in fact there isn't one.

"la la la my fingers are in my ears I can't hear you!" is the best you got? You seem to admit iouring exists, isn't POSIX, but what? You just don't believe it's faster now? Bull shit.

> You've already provided an example. The Linux kernel provides both select, a posix interface, and io_uring. Nothing about providing posix interfaces precludes providing specialized interfaces as well.

If you think a reimplementation of Linux is "easy" you need to have your head examined: Microsoft has near-infinite money and eventually gave up, deciding POSIX was too hard, and just implemented another hypervisor.

"Two processes! Not two threads!"

And I wrote "thread or process" because, architecturally regarding IPC, the distinction is unimportant.

"How would you know?"

Due to the inaccuracy of your analogy, with regard to actual systems interfaces.

"You seem to admit iouring exists, isn't POSIX, but what? You just don't believe it's faster now? Bull shit."

io_uring is not faster than shared memory via mmap.

"If you think a reimplementation of Linux is "easy" you need to have your head examined"

I think now we're touching on the limits of your experience and expertise. Implementing select(2) on Linux is indeed quite simple. Have you ever read through select.c?

Have you ever even implemented a posix subsystem? I have.

"Microsoft has near-infinite money and eventually gave up, deciding POSIX was too hard, and just implemented another hypervisor. "

This isn't how things work on Windows.

As an aside, I've noticed that your ignorance has shifted into rudeness. Don't bother to respond again. This conversation is over.

Kids today: want to come along, throw away everything they don't understand, and rebuild what was already there but worse. Accomplishing negative impact.

https://fs.blog/chestertons-fence/

I think this is a human thing, not necessarily just kids today.

Statistically, I think it'll always be more kids doing this, regardless of day of the week or year of millennia.

I think it depends on your experience and what you have gone through, in my case how many of the decisions I've made that looked excellent to me back then, turned out to be okish at most.

Can we actually improve the POSIX shell?

Can we reform it, perhaps into a "D-shell," that is an LR-parsed language?

AWK had a yacc grammar for many years. Why can't this be the language of the shell, at least for the sake of maintainability and comprehension?

https://archive.fosdem.org/2018/schedule/event/code_parsing_...

The problem with making a shell more like a "real" interpreter REPL is the purpose of a shell: Running arbitrary programs not known to the author of the shell, such that any unknown word in a shell script is probably the name of a program as opposed to a misspelled reserved word or variable name. Those programs having arbitrary command-line arguments and having to operate on files with arbitrary names is another complexity.

It has been done. See xonsh a python shell which manages the distinction you mention.

Also see scsh and rash for scheme

Fish for non posix but sane syntax

It's interesting to see how so many things grow into each other. A shell is what you say, but there are many efforts to change that. Bash has its magic autocomplete which knows about options of many, many programs. Powershell has an object-model with knowledge of many of its commands.

> Can we actually improve the POSIX shell?

PowerShell?

No, that won't work.

  $ ll /bin/dash
  -rwxr-xr-x 1 root root 85368 Jan  5 09:52 /bin/dash

If you can make Powershell in a binary that size, then you have a valid point.

But why? Storage is much cheaper today than it was in the 80s. While I myself don’t like huge Electron apps, trying to save every byte of storage is simply not worth it.

Embedded applications on smaller systems are why.

The POSIX shell is common, from the largest supercomputer down to the smallest ARM.

I don't think that there will be any new developments covering that full scope.

Embedded systems don't exclude PowerShell from being a replacement for POSIX shells. Maybe embedded systems should not strive to be POSIX.

Many embedded systems hardly have any shell, and even when they do, it isn't bash (which isn't POSIX anyway), rather a tiny subset with the basic tooling for maintenance.

Part of the process of erecting your own identity, both individually and as a generation, involves flatly refusing and denying the practices of your immediate predecessors (eg: your parents' and their generation of men, source code written by programmers before your time, etc.).

It's stupid and almost always leads to unintended (and usually negative) consequences, but it's probably something strongly ingrained in our instincts because it rears its ugly head every time there's a generational turnover.

If you do not understand why this happens, do not call it stupid. Go away and think. Then, when you can come back and tell me that you do see the use of it, I may allow you to call it stupid.

It's stupid in the sense that it's a waste of time and resources.

Sooner or later after you've had your fun refusing and denying your predecessors, you realize why they do things the way they do, become wiser, and go back and undo and redo all the fuckery you caused with your refusing and denying.

It would be nice if we could get that realization without first having to fuck everything up, y'know?

Even in this description you use the words "you realize" and "become wiser" and then describe it as "a waste". It's not a waste, it's just the cost of education.

As the saying goes: "good decisions come from experience, and experience comes from bad decisions". It would be great if we could all come out of the womb with enough education and experience to function as responsible adults in modern society, but we don't. Society (rightfully) allows people some leeway in doing dumb shit so that they can learn firsthand why it's dumb.

The experience is absolutely not a waste, don't get me wrong. What I'm calling a waste are the time and resources spent undoing and redoing everything.

It's all cute if this just happens during our childhood years, where most actions hold no real consequence. It's when this extends into the real world with very real consequences that it gets really wasteful.

Well we do this for many situations where it does actually matter. Drivers license requirements force people to "do it right" the first time, you can't (legally) fuck around driving a car on public roads until you eventually figure it out. Airline pilots and nuclear power plant operators have even stricter requirements.

Licensing and training has a very real cost as well of course, and for many topics the cost of forcing everyone to get a license would outstrip the benefits. So you either have to spend on training, or you have to spend to fix the mistakes of untrained people, or a mix of both. There is no way to bring the waste to zero.

I'll be glad to pillage your stagnant empire meanwhile :)

You forget one thing ... You also discover the 20% of manure that isn't true, or doesn't apply to you.

Surely if you don't understand why it's stupid, you need to go away and think about why it's stupid, once you do see why it's stupid, I may allow you to challenge people calling it stupid.

Great article.

Discussed 13 months ago:

https://news.ycombinator.com/item?id=30070757 (142 comments)

Some variation on this has been consistently brought up for going on a decade now. At this point it's a HN meme.

Its opposite is "cargo culting".

I like how Chesterton’s Fence example actually is a huge pro to not letting unquestionably the fence in the middle of the road.

If there is no documentation on it, no-one around to pro-actively explain why it’s there, then the most economical way to see if it had any positive value is to stash it and see what happen.

That doesn’t mean it’s always the most rational thing to do, of course. Throwing POSIX away doesn’t qualify with a situation where there is a lake of provided documentation or a missing obvious purpose.

The fallacy is well confined in these two sentences:

>However, before they decide to remove it, they must figure out why it exists in the first place. If they do not do this, they are likely to do more harm than good with its removal.

Actually, assuming you don’t know consequences of removing the fence, you also have to assume that you don’t know consequences of letting the fence in place, which might just as well turn to a dramatic situation. What is for sure, if nothing around can explain its purpose, is that it was placed here by people to negligent to induce the right clues about it. So, who know to which level their laxity might have reach?

Remove the fence unless someone know why it was put up in the first place, and the rational still hold.

Story time: at my previous job we had an old server called "the black box" which hosted some old cronjobs and various services from about a decade earlier. There was no documentation and everyone who had worked on it had left years earlier. The ops team dutifully kept it online but otherwise didn't touch it, and no other team owned it. In short, it was the ideal candidate for your "reverse Chestertons fence" strategy. At some point we had to delve into what was actually on it, and it turned out that both company payroll and a service responsible for about 50% of company revenue was on that box. Simply removing it to see what happened would have easily cost a few million in revenue, even if it could be reinstated from backup a day later (and I'm doubtful if we could have restored it at all without serious data loss). Now obviously having such a server is a bad practice in the first case, but if you find yourself in such a position "just kill it and see what happens" is definitely not the most economical option in all cases.

In the case of POSIX it might hold, since probably the worst case is that some volunteers throw away a few years of their life hacking away on yet another OS that will never gain traction. Humanity spends a lot more time on a lot less worthy endeavors.

Why are you putting the onus on someone else to provide clues that you understand?

You still don't escape the requirement to understand the problem before fixing the problem. Even if the explanation for the current solution is lacking, you still need to understand enough to articulate why it's lacking.

It is completely crazy to expect people to "pro-actively explain why it's there" -- how do you see this? A plaque at every fence with explanation? A person standing by fence every day, just in case someone wants to tear it down? Do you expect each shared object to be labeled and documented?

Whomever wants to tear the fence down has to do basic research -- visit city hall for the plans, talk to old-timers, check the old newspapers, etc... If they do it, this qualifies for "understanding". And yes, maybe the fence was put by Crazy Old Bill who was paranoid about neighbour's goats... then yes, tear it down. But make sure it is really him first, and not some better reason.

> assuming you don’t know consequences of removing the fence, you also have to assume that you don’t know consequences of letting the fence in place,

Note that at some moment there were no fence, and yet someone has expended non-trivial effort to put the fence up. This means someone had a reason do to so, so whomever that past person was, they knew more than you did right now. Therefore, unless you believe that you are smarter than most other people, you should defer to their past decision.

(And if you believe that you are smart that most other people, and yet want to tear down the fences you don't understand... I have bad news for you :) )

>A plaque at every fence with explanation?

Well, for every fence at the middle of a path that have for example a a commemorative value, that seems a good start. And "warning do not remove" is a rather common thing on places where a danger is expected otherwise.

>Whomever wants to tear the fence down has to do basic research -- visit city hall for the plans, talk to old-timers, check the old newspapers, etc...

I think that qualifies in "someone know why it was put up in the first place, and the rational still hold".

>Therefore, unless you believe that you are smarter than most other people, you should defer to their past decision.

I don’t think smartness matter that much here. It’s all about blindly rel invariably in a cargo cult fashion about everything or keep critical thought vivid when it seems possibly relevant. Someone can be extremely smart and still follow rigorously the cult of the day.

-----------

Some thought experiment now. :)

Say you won a new house, congratulation!

In the middle of the garden, crossed by a path, there is a fence with no door, easy to circumvent, serving no apparent purpose. The planes you where given with the house, duplicate of those from the city hall, don’t mention it. Now, there might many explanation for this situation. One possible rationale would be, the fence used to be larger, possibly even forming a closure some time in the past. But no one know. Will you let let the fence disrupting the path in the middle of the garden? Will you call some expert to diagnose the soils, just in case it might collapse would someone remove the fence?

Also, in the middle of the living room there is a thick wall up to the ceiling, with large space to pass on both side. It’s not mentioned in any plane. Having a unified big living room would be fare more nicer to you. Will you destroy the wall without delay? Or call an expert to determine if this is a bearing wall first, without which the house would collapse?

So this is really largely dependent on context, as usual.

> Actually, assuming you don’t know consequences of removing the fence, you also have to assume that you don’t know consequences of letting the fence in place, which might just as well turn to a dramatic situation.

That assumption reads as false to me. There should generally be historical data around the costs and consequences of maintaining the fence. E.g. if you're shutting down a server, it's generally easy to figure out how much the hardware costs, how many man-hours you spend on maintaining it, etc. None of that requires knowing why the fence exists.

Those costs and consequences tend to be small, because large costs have to be continually validated as a good use of resources so they have a living memory of sorts.

> What is for sure, if nothing around can explain its purpose, is that it was placed here by people to negligent to induce the right clues about it.

Again, I don't think this is sure. I've seen lots of cases where docs probably did exist at some point, but the writers left and successors lost the knowledge of where the docs are or failed to migrate them to a new system or etc. They're lost to attrition, not to negligence.

> Remove the fence unless someone know why it was put up in the first place, and the rational still hold.

It doesn't, because it's trading known risks for unknown and unbounded risks. That's usually a bad trade, especially when the known costs tend to be small. Nobody wins a promotion for causing a million dollar outage while trying to get rid of a $50/month maintenance cost.

Ok, these are all good points, thank you for sharing it. Just as the ones by WJW.

More broadly my underlying point was more as "don’t blindly follow the herd obedience".

If there is a small black box in multi-million generator stream process, no one of course want to be the guy unplugging it just to see what happens.

On the other hand, if really no-one knows the real impact of such a black box, given the resources at play, risk management should induce an audit and possibly a replacement management. To my mind that sounds very different from "no one know why this thing is there, those who dare to ask will be chastised with a burden of proof to carry alone".

I can't upvote you enough.

Around here, on HN, Chesterton's fences pops up so many times as some type of ageless nugget of wisdom.

But another way to look at it is that it's just the lazy guy's excuse to allow bit rot. Because, let's be honest, we only care about Chesterton's fence in the context of software development.

Who among us did not work with software with dark corners, with files with no commits in the last 10 years, with playbooks that people just go through mechanically, because there's no one from the "old guard" who remembers why things are the way they are?

Who would not love to live in an alternate reality, where each Chesterton's fence was removed at the first opportunity, rather than left there to burden future generations of unlucky toilers? Where code is clean, documentation is up to date, tests complete, colleagues who know what's going on? Well, that alternate reality does not exist, of course.

But at least let's stop thinking we are wise whenever we postpone deprecating something we don't understand, just because some semi-famous essayist came up with a nice figure of speech one hundred years ago.

It's probably worth noting that the author is in his 50s and is fairly familiar with POSIX (he has been involved in the FreeBSD developer community for decades).

which makes this garbage even more indefensible.

(smirk) You've noticed that there are lots of kiddies, getting all hot under the collar, who haven't noticed that a counter-example does not refute the global mindset that is Posix ;)

That's been the default for the whole software industry for the last 20 years at least.

Reinventing things without learning from the past - and if you point it out people gets so defensive.

> Accomplishing negative impact.

This opinion piece is the epitome of a negative impact work. It's easy to accuse random things of being all wrong, but it's far harder to actually present something you feel is right.

Not every criticism of something has to be accompanied by a better solution. Pointing out that something has shortcomings is the first step. It’s perfectly fine to say, “That person’s singing is awful,” without also adding, “They need to work on their breath control.” You don’t need to be a great singer to know when someone’s doing a bad job at it. (And many people who are great at what they do are terrible at teaching others to do it, anyway.)

Criticism without suggestions for improvement is almost always useless.

Then why do company constantly ask me to "score this phone call", "score this driver", "was this person helpful?", "what did you think of this recommendation?", "could you spend some time to answer this questionnaire what you thought of that widget you bought"?

So to me it seems like criticism without suggestions is very valuable, it is called data. Suggestions from amateurs are usually bad and misleading, just highlighting a problem so that the expert can come up with a solution is the most efficient way to work in most cases.

I suspect a lot of that data provides very little value.

If you don’t know why the customer was unhappy, what do you change? You can fire the customer service rep, but then what do you do when the next one gets reviews just as bad because they are following the same script?

"This food is terrible" merely needs to be true, and anyone can say it, even people who aren't chefs. This is a ridiculous barr that only sorta kinda sounds reasonable if you just want an excuse not to like the message or the messenger.

> "This food is terrible" merely needs to be true, and anyone can say it, even people who aren't chefs.

If you bitch about how "this food is terrible" but you do nothing to make it more palatable or order something else and instead just stand there eating while complaining and reordering and asking for seconds... The food is not terrible, and you just enjoy bitching.

I would have warned you about the pot hole that destroyed my car instead of letting you get nailed by it too, but that would have just been bitching about something without fixing it, and I know you don't have any use for that.

> Pointing out that something has shortcomings is the first step.

Talk is cheap. If even that same critic feels that his solution to his problem is not worth pursuing, the criticism adds nothing and creates no value. At best it's nitpicking about stuff that falls in Eisenhower's "not important"/"not urgent" box.

Do you need to be a chef to realise some food is burnt?

and the old folk that created all these things, don't bother to explain them, then get uppity when someone new comes along and doesn't think their creations are perfect.

2 sides of the same coin.

Do you really have to call out the entire JavaScript ecosystem like that? :)

https://www.jwz.org/doc/cadt.html

It's fun to build stuff. It's not nearly as fun to use and maintain what other people have built. Why should all the fun be for the people who just so happen to have been born decades earlier? The point of life is not to maximize efficiency in all respects, and people make emotional choices to achieve personal fulfillment.

There's a difference between (re-)building stuff, for fun, learning purpose or even to use in production in your own projects, and dunning-kruger demanding (exaggerated) that existing solutions that provide stable APIs for multiple OS are thrown out, causing a massive amount of forced work (i.e., not fun) to others while getting a worse end result.

Also, it can be fun to explore existing ideas and build upon those too - nobody forces you to maintain anything existing "for fun".

My problem with this is that I feel like we (people coming along later) have three options when it comes to designing systems given the existing state of posix / linux / the web / etc:

1. Build inside the ecosystem as it was designed. Eg, make a static webpage using HTML. Make a linux binary using the standard linux tooling + apt (or whatever). This is almost always a good idea when you can, but sometimes the ecosystem is a bit shonky, or misaligned with what you're building. Eg, I want my program to have a fixed execution environment and instead I have debian and macos. I want a react-like API but the browser doesn't provide that. Etc.

2. Build a new thing on top of the ecosystem as it exists. Eg, build docker on top of linux. Build web frameworks. Invent the web browser, or couchdb views, or electron and build your software on top of that. The problem with this approach is that if each generation layers their own rubbish on top of the underlying layers then software will keep getting slower, more complex and more buggy over time.

3. Change the underlying system to work how you want it to work. Instead of using docker on top of ubuntu, replace ubuntu with nixos. Instead of using dpdk for higher storage performance, add io-uring to the linux kernel.

I think I would prefer it if more people did option 3 rather than option 2. I think there's lots of ways modern computing systems could be better. I'd rather if linux gained the ability to run binaries from the web (for example) than we invent a new kind of binary in userland (docker containers) and run those. I don't want people in 100 years to still be using linux (with all the bad decisions it contains), but with 10 more layers of different generations' ideas stacked on top. (Oh, its linux - but with docker, and then most people run this specific container to run languageX, and that lets us run electo2050 apps, which in turn you can use to run IRC. It only uses 50 gigs of ram. So small and light!)

Option 3 may require us to throw out or deprecate parts of the POSIX API. That may be overdue.

You present 3 independent Choices, but surely these are stages.

1. Can you do X with existing tooling?

If not

2. Can you add a tool to your set to do X?

If not

3. Get a new tool set.

2 should also serve as testing to show the utility and iron out the links of things that make it to 3. Jumping straight to 3 likely means you end up with an ill thought out solution that soon needs another 3.

Sorry; that’s not quite the point I’m making. The difference I want to emphasise between 2 and 3 is whether the new tool is layered on top of the existing tool, or made by modifying the existing tool. If you want a new approach to storing data, do you make it by writing a new kernel module, or by doing some new thing in userland on top of the filesystem?

All of docker’s features could be built into the Linux kernel. And it would probably be better as a result, and have ancillary benefits to existing Linux programs. But docker was built as a separate userland tool instead.

Building a new thing is usually easier, since you don’t need to dive into the politics of an existing community. But if each new generation of software engineers builds new software by layering on top of the old stuff, then we end up with layers on layers of cruft and our computers run slower and are more buggy as a result.

The POSIX apis aren’t very good in lots of ways. With a lot of complex effort in userland I can make a database with atomic write semantics. But maybe we should just add atomic file operations to Linux instead. In the long run I think it would be safer and result in faster programs in most cases.

Yes. Docker has proved there's utility in what it does. Assuming kinks in the implementation have been ironed out, now is the time to start thinking about making it a kernel module.

If you just put every half thought out, poorly implemented idea straight in the kernel, you'd have a 5 petabyte kernel.

While complaining that Git is haaaard.

Well git is hard tho

It's easy if you always do exactly the right thing in the right order and never make mistakes, but if you for example want to undo a git merge[1], god have mercy.

[1] https://opensource.apple.com/source/Git/Git-26/src/git-htmld...

In doubt, I create a new branch from the main one, and test it goes when merging the one holding changes. It something goes wrong, at least the main branch is unaffected.

Of course, you must already have some intuition that something could go wrong. :D

Reflog lets you see where HEAD has traveled from. Just go back in time. This isn't actually a hard problem to solve.

If that sounds difficult just create a branch prior to merging so that you know where to reset your failed branch to.

They are trying to solve this problem without rewriting the branch.

It's not that hard, it's actually easy. Very Easy to get into a devolved state where the shortest path back to sanity is to clone the whole repo from scratch.

Yeah, KV is GNN's vehicle for excessively controversial (i.e., what I might call "bad") takes.

I don't think he's wrong that the POSIX synchronous IO model isn't a great fit for modern hardware, though. He doesn't really go into it much but (Windows) NT's async-everything IOCP model really seems to be the winner, as far as generic abstractions that have stood the test of time.

> I don't think he's wrong that the POSIX synchronous IO model isn't a great fit for modern hardware, though.

Opinions don't really matter. Solutions to real-world problems do.

To me, the fact that no one up to this day felt that the issues in this area was significant enough to warrant a fix or an alternative tells me that it's noise about nothing.

> To me, the fact that no one up to this day felt that the issues in this area was significant enough to warrant a fix or an alternative tells me that it's noise about nothing.

Do you not view io_uring or Windows IOCP as alternatives? Or the plethora of other non-POSIX IO extensions in Linux, such as splice and sendfile? SPDK and DPDK? There’s also Google’s Fuschia operating system, which is not Posix. I think it’s pretty clear that people are investing in a variety of alternative approaches.

so what you're actually saying is that the POSIX elephant isn't (and hasn't been) actually standing on our necks after all?

I mean if what they want is a better standard, so there can be a dozen compliant operating systems we can all compile our stuff for, then maybe.

To look at things from another angle, what if having had POSIX all these years actually made things more uniform, in the sense that we didn't have variance between systems as large as e.g. between a BSD Unix and a Lisp machine? What if that were our choices today. As it is the largest mainstream departure is maybe Windows.

None of the things mentioned are POSIX _or_ portable.

> Do you not view io_uring or Windows IOCP as alternatives?

I don't simply due to the fact that no one ever bothered to put something together on UNIX-like systems that resembles any of those options.

If anyone felt that those were worthy ventures, in these last few years we would already have seen something resembling that effort. But we haven't.

> no one ever bothered to put something together on UNIX-like systems that resembles any of those options.

What do you mean? io_uring is a Linux kernel interface, and there are several userspace projects that are building upon it.

> To me, the fact that no one up to this day felt that the issues in this area was significant enough to warrant a fix or an alternative tells me that it's noise about nothing.

yes, let's just ignore the mountains of people who end up literally reimplementing complete network stacks in userspace to handle events in a modern way

> yes, let's just ignore the mountains of people (...)

Who are they, exactly, and why haven't anyone from that mountain thought it would be a worthy investment of their time to pull it off?

I see you complaining that people have to implement things, but why has no one ever implemented the thing you perceive to be so invaluable?

Because usually they are implementing a subset of the network stack that meets their needs for communication within their internal network. Porting the entire kernel stack that actually handles all of the complexities and quirks of the open internet is a larger task than they needed to tackle.

People/projects can and do implement userspace networking stacks. For example, gVisor implements its own userspace networking stack [1] isolated from the kernel networking stack.

[1]: https://gvisor.dev/docs/user_guide/networking/

Yeah, I agree that's an area that needs work. I admit I didn't bother to read every word, but I did search for "async" to see if he mentions aio(7) to complain about it and I saw that he didn't.

Don't get me wrong, there are lots of room for improvements in POSIX IO interfaces. For example, POSIX doesn't define any modern event systems (epoll, kqueue, etc). But what's the result? We use libevent/libev.

This sort of seems like the result he's asking for in diverging from POSIX, which we ... already have. There are a lot of pretty great non-posix standard interfaces available too!

Concurrency is hard. If you don't have anything better to do but wait on a single file operation (which, in a CLI tool, you might not), then a synchronous call is just fine. If you do have multiple I/O operations to issue at once, that can still be synchronous with writev().

Most forms of async programming are also unstructured, which is bad for correctness, but also for performance since it can lead to priority inversions.

NT's async-everything IOCP model really seems to be the winner

The author talks about plumbing to data processing ratio. Let’s take hundred average programmers, let them plumb IOCP and see how many of them can even get to the “data” part.

I really disagree with the sockets bit. Sockets are designed for networking, with a particular focus on IP. Not IPC.

I for one think that the 'absurd thing' is that IPC is not built into OS as a core feature. That, and process isolation. Both sockets and shared memory are quite problematic and the challenge of 'true IPC' that works nicely with threads etc. is real.

unix domain sockets are designed to be used entirely locally.

and files are one-way sockets. They lied when they said "everything is a file", but it's almost true that (for Posix) "everything is a handle."

Win32 is POSIX! => "everything is a handle."

> IPC is not built into OS as a core feature

POSIX and its derivatives build IPC into the OS as a core feature. In particular, POSIX is built around memory mappings and file descriptor inheritance, which means it is extraordinarily easy to make processes communicate.

I honestly have no idea what you mean by this statement.

Unix domain sockets are (1) fast, (2) primitive (just a file), (3) widely available, and (4) can be used between multiple processes extremely easily (use SOCK_DGRAM).

I honestly have no idea why someone would think that Sockets, which are designed for IP networking are an ideal solution for IPC, or why someone would think that 'shared memory' is sufficient for sharing information when it's really only part of a solution.

If it were easy than everyone would be doing it the same way and there wouldn't be a discussion about it.

With shared memory in particular, the issue gets tricky around signalling and locking aka indicating to other processes when new info is available, and when another process is accessing the data.

If anyone took just a moment to design an API that was user/developer centric, and worked back from there, it would look materially different from what is provided today.

> Sockets, which are designed for IP networking

Sockets aren't designed for IP networking. They date back to a time when it wasn't even clear TCP/IP would win vs. IPX/SPX. Or DECnet. AF_UNIX sockets have things that IP sockets can't even dream of doing (the very poorly named SCM_RIGHTS). And sockets don't really cover IP networking all that well, e.g. MPTCP and SCTP shenanigans to get multi-address & multi-stream connections.

That said, both sockets and mmap are low-level APIs. A higher level "user/developer centric" API indeed looks different.

> With shared memory in particular, the issue gets tricky around signalling and locking aka indicating to other processes when new info is available, and when another process is accessing the data.

POSIX has mutex and condition variable, is this not enough to do those things?

As far as I know mutex is not a signal that data is received, which would require some kind of threading / interrupt.

Moreover, it's a bit of a secondary thing.

The API would look something like a call function with a receive function/lambda on the receiving side, with memory possibly being 'handed over' or whatever. And so the level of abstraction would be slightly higher.

And it would be a no-brainer, well established thing. Searching for how to do that today and it's always complicated.

I’m afraid I don’t understand. If you want one process to signal to another that data is available, you have the receiver block on a condition variable. When the sender writes its data it signals the condition variable. This is all handled by the kernel, no signal/interrupt is required. Or if you don’t want blocking, you can have a shared ring buffer, with the two processes interacting with it atomically.

Which is how dbus and the efforts to get it in the kernel were born.

Completely agree. In particular, POSIX is built around the inheritance of file descriptors by their children, which means that it is extraordinarily easy to have sockets going between multiple processes. Moreover, it's entirely possible to send file descriptors over other sockets (SCM_RIGHTS). POSIX has robust IPC. I'm currently messing around with IPC on Windows... and wow, at the end of the day, even in 2023, UNIX et al are simply more advanced than Windows. It's unfortunate there's been absolutely zero groundbreaking discoveries or inventions in this field (OS dev), but the idea that we should throw away the state of the art simply because is just silly.

POSIX IPC has withstood the test of time. There is no other system that offers as rich a set of primitives.

Can you be more specific about how POSIX is more advanced than Windows (NT kernel)? Or a primitive it offers that NT doesn’t? Examples?

Windows provides most (all?) of the same mechanisms, and a number that POSIX does not.

https://learn.microsoft.com/en-us/windows/win32/ipc/interpro...

Example Windows concepts that Unix doesn’t have (AFAIK):

Transactions on named pipes (request/response): https://learn.microsoft.com/en-us/windows/win32/ipc/transact...

Parents can control which handles child processes inherit: https://learn.microsoft.com/en-us/windows/win32/ipc/pipe-han...

Remote Procedure Call (RPC) IPC: https://learn.microsoft.com/en-us/windows/win32/ipc/interpro...

ACL-based permissions on objects that kernel APIs operate on, including pipes. (ACLs are a much more sensible way of specifying permissions on resources than owner/group). https://learn.microsoft.com/en-us/windows/desktop/SecAuthZ/a...

Since windowing is built into the NT kernel, it also has support for powerful clipboard operations as one crucial type of IPC in a graphical environment (see 1st link).

Windows I/O Completion Ports (IOCP) also provide a high-performance way to implement kernel-managed asynchronous I/O operations which has no parallel in POSIX: https://learn.microsoft.com/en-us/windows/win32/fileio/i-o-c...

I’ve been developing professionally on Linux for >15 years, and while I do like its simple aesthetic, the consistency and power of NT kernel APIs are something I miss.

Sure there are, one only needs to actually bother to learn about the history of computing and other platforms, to validate that.

Reply on news.ycombinator.com