Comment by klabb3 | Hacker News Reader

Comment by klabb3 | Hacker News Reader

Parent Full thread

While I don’t doubt that the author has good reason for their opinion, the reason is handwaved here, no? Where is the tech debt? In the kernel only? Does it affect user space? For who is it more or less composable?

I personally think ”async IO” (a term I don’t love, but anyway), should be agreed upon and settled across OSs. Not necessarily exact syscall equivalence, but the major design choices needs to be the same. For instance, who owns the buffers (kernel or user), and should it be completion or ready-based event systems?

The reason I think this is so important to agree upon, is that language runtimes (and other custom runtimes like libuv) need to be simplified in order to allow for healthy growth. Any language that doesn’t have cross-platform support for ”async IO” is completely nerfed, increasing the barrier of entry for things that otherwise would be ”graduate level” projects, into deep OS/kernel knowledge that is hard to find outside FAANG and a few other spaces.

The claim is silly and unfounded. If anything, kqueue is the interface that pollutes userspace less. You have a single kevent call that is used both for waiting and registering events represented by (filter, ident) tuples. All of the data related to the event is contained in a struct that’s passed between the kernel and the user. For non-fd events (EVFILT_PROC, EVFILT_TIMER, EVFILT_SIGNAL), it is much more straightforward to use compared to the Linux way where you need to keep track of one more resource, read specific binary data from it, have specific flags, etc.

The article reads more like pr damage control than an actual complaint. epoll has a reputation as a fragile, hard to correctly use interface. and kqueue as relatively simple and sane. So the question always is. why didn't linux just adopt the already existing kqueue instead?

Sometimes it seems that linux does end up with more than it's fair share of technical excellence coupled to a bad interface. iptables, epoll and git come to mind.

This other article has been discussed in the past on HN:

https://idea.popcount.org/2017-02-20-epoll-is-fundamentally-...

https://idea.popcount.org/2017-03-20-epoll-is-fundamentally-...

Because it discusses epoll in much more detail, it is far more convincing than the parent article of this thread.

The conclusion of that article is that how to use correctly epoll is not at all obvious and it has some pitfalls that are not easy to avoid. Therefore epoll seems to be more affected by a serious technical debt, i.e. by a less than good design of the original API.

If you come from the starting point that Linux is correct and everything different is wrong then you get blog posts like this.

The author thinks that musl, Alpine, and Linux are right and everything else is wrong.

* https://ariadne.space/2022/03/27/the-tragedy-of-gethostbynam... - musl's DNS resolver doesn't support TCP lookups, but it's your fault for expecting the standard DNS lookup functions to work. You should change your program to use a third-party library for DNS lookups. This post is a couple years old; musl now supports TCP lookups.

* https://ariadne.space/2021/06/25/understanding-thread-stack-... - Alpine has a stack size that's substantially smaller (128k) than other widely used OSes (512k+). Your program may work everywhere else, but you're wrong to assume a reasonable stack size. Here's how to jump through a hoop and make it work on Alpine.

Same person that both contributed the code of conduct to Alpine and later got caught bragging on twitter about having bullied people out of the project.

The work of someone who did not ask themselves the necessary questions, but now its some years later and things have changed.

The second article is not saying that Alpine is right, only that it behaves differently. "Reasonable" stack size is pretty subjective - many workflows won't care about stack allocating megabytes at a time and could save RAM from having a smaller stack size. The article is pretty informative with workarounds for this situation. There's no need to attack the author about this, especially from a throwaway.

Sounds like an asshole.

> While I don’t doubt that the author has good reason for their opinion, the reason is handwaved here, no?

I came to the comments hoping to get more explanation. I was waiting for the full explanation of the technical debt but then the article just came to an abrupt end.

I wonder if the article started as a short history and explanation of differences, but then the dramatic headline was chosen to drive more clicks? The article could have been interesting by itself without setting up the “mountain of technical debt” claim in the headline.

I know ariadne well enough that I'm pretty confident that the headline will have been chosen out of annoyance rather than click maximisation.

That doesn't mean I agree with the conclusion (I am ambivalent and would have to think rather more before having an opinion myself) but I'm reasonably sure of the motivation nonetheless.

I fully agree.

Synchronous I/O looks appealing for beginners and simple projects. It seems like the simplest path for things where I/O isn’t the primary focus.

Then one suddenly finds themselves juggling ephemeral threads to “background” tasks or such. The application quickly becomes a multi-threaded monster and still suffers responsiveness issues from synchronous use.

The Tower of Babel created from third party runtimes further adds to the mess even for those who try to do better.

> Synchronous I/O looks appealing for beginners and simple projects.

Yes. And.

Synchronous I/O is suitable for almost all projects

When it is unsuitable it is catastrophicly unsuitable,

Synchronous I/O is actually better than async for many programs, partly because almost all async systems were designed without thinking about priority inversions and can't resolve them properly. If you have futures, you have this problem.

Since the NT kernel is async throughout, it solves this with periodic priority boosts for non-running but runnable threads.

I swear, the kernel designers just need to look at NT :-P But we're well beyond a complete rewrite of old kernels, sadly.

That can work but I think it's cheating. Especially if your scheduling is tied into power management or you have high/low power cores, then boosting the low priority threads will waste power.

Forgot to say, the correctly designed ones are called "structured concurrency".

The entire unix operating system is designed to abstract away async problems and turn them sync so you can write sane code (if, else, for).

In my experience simple blocking code is incredibly misunderstood. Most programmers stop their thinking at “blocking is slow and non blocking is fast!”

> The entire unix operating system is designed to abstract away async problems and turn them sync so you can write sane code (if, else, for).

That only works for workloads that can be independently parallelized. For stuff that cannot be - which is the majority of consumer workload - you're in for a world of pain with locking and whatnot other issues.

No. It’s almost all the time that your computer has more than one thing to do;

- your browser uses a process per tab - your web server responds to multiple requests in parallel - your web app is going to launch other programs - your desktop ui is going to have a compositing server separate from each application - your compiler is going to work on multiple source files

> In my experience simple blocking code is incredibly misunderstood.

I don't think so. You can't even implement a full-duplex protocol properly with sync IO since reading and writing can't happen at the same time on the same thread. Splitting them to different threads also won't help since accessing shared state requires synchronization with e.g. mutex that kills again simultaneous reading and writing.

How do you propose to implement duplex IO? And what’s the example case?

With threads the reading and writing can happen simultaneously. The synchronization is only on whatever data you need to share.

Note that if reading and writing are dependent, then single thread actually makes sense because you can’t do one without the other.

Otherwise the pattern of logic you will execute for reading or writing is completely different, which is why threading is the perfect abstraction.

I implemented split IO with io_uring, epoll and eventfd and my own thread safe comm pattern

You can write to an eventfd to wake up io_uring listening on an epoll instance.

The differences in design reflect differences in use case assumptions, which in turn reflects that different OS tend to be used for different workloads.

Broadly speaking it is difficult to use async I/O well without understanding the implementation details. It is a sharp tool for experts, not magic “go faster” juice.

Which is why parent is calling for standardization, so that people don't usually need to be experts. I reject that there are a diverse range of async abstractions that are all justified such that a given OS only supports some well. More likely, that means the designs are poorly done. Although I can't say anything if it's for backwards compatibility, how I loathe and crave it.

The bigger issue is that no OS was designed for expansive and flexible async functionality. The implementations reflect practical limitations of remaining compatible with the rest of the guarantees the kernel makes. I would love a portable standard, I’ve been doing async for a long time and I would benefit greatly, but I also understand why it hasn’t happened.

A good standard would likely require every OS to materially redesign its internals in a way that would have backward compatibility issues for the other users of the system. I am not against the idea since I would clearly benefit personally, but I understand why the tradeoffs make it unlikely to happen.

Even going from io_submit to io_uring just on Linux changes the way you need to architect your software, aside from API changes, and that is the same kernel. It deeply touches the behavior of the system.

> The bigger issue is that no OS was designed for expansive and flexible async functionality.

The NT kernel was designed in this very way. All I/O in the NT kernel is async. It was one of the design principles.

a good computer system would allow the application programmer to write their code in a synchronous and single threaded manner and all performance optimizations with respect to multiprocessing (and associated resource sharing) would be handled automatically.

I'm not sure providing that level of complex logic and scheduling is really a proper task for OS kernels. It's essentially what you get from green threads / coroutines which are provided by numerous runtimes and frameworks, but those have to be translated to the various different async IO facilities in order to be portable between OSes.

Disagree. Scheduling is exactly the type of thing the OS should be doing. It’s (a) extremely easy to get wrong and cause starvation and other problems (b) very complex and large in general and most importantly (c) the OS is the only component with birds-eye view across all processes and hardware (like Numa cores). Every major OS is already doing it, but not for async IO. So it’s absolutely reasonable to maintain this responsibility during the transition to async IO.

This goes back to my initial point. We should not need a team of PhDs and kernel hackers to do hello world. The old sync syscalls have largely uniformity across platforms, but the async ones don’t. This means, in practice, that most applications, runtimes and thus users, don’t get to enjoy the benefits.

Counterpoint: the OS should indeed do its job of scheduling, but applications sometimes know better (the ones specifically implemented by programmers who know better). The OS and application must cooperate to do big jobs like scheduling. See Linux's clunky NUMA lack-of-awareness and transparent huge pages blunder for examples of how sole OS control is lacking. This is not fundamentally a Linux implementation issue: the OS does not always know best. The other extreme, giving applications control, is obviously wrong too, like cooperative scheduling being infeasible for general purpose systems. The OS should have strong defaults but also empower applications (or rather, their programmers) to try their utmost if they want. The OS is not the end all, be all. It is the platform on top of which applications gain valuable power, and constraints. That is basically by definition.

Ack.

> but applications sometimes know better (the ones specifically implemented by programmers who know better)

Sure, and I don’t mind it. But right now to get basic modern performance you need that knowledge. That’s my objection. I just want more projects to be able to benefit.

> See Linux's clunky NUMA lack-of-awareness and transparent huge pages blunder for examples of how sole OS control is lacking. This is not fundamentally a Linux implementation issue: the OS does not always know best.

I’m not familiar. But, I have a question. Do you think it’s possible to create a middle ground, where applications can specify a small (ideally simple) params in order to allow the OS to make meaningfully better scheduling decisions (as opposed to relinquishing responsibility entirely)?

The biggest roadblock is, as always, backwards compatibility. The second biggest is getting people to agree to standardization and documentation, solidifying the expected base of knowledge. All in all, I don't have much hope for mainstream OSes to get these improvements. But in theory it could be done. For certain workloads, applications could even just make minor adjustments like your suggestion of scheduling policy hints for great gains. This is especially about being more sympathethic about context switches, which can embody better core/NUMA socket locality, better awareness for interactivity or batch processing, better performance/efficiency core allocation, etc.. So in general I would say we could get quite far without much application programmer or user intervention.

This is wishful thinking. It would be nice, I agree, but. Even "guaranteed lower bound of optimization" for general purpose programs is off the table. Our tools for programming should be such that easy problems are easy to solve, and hard problems are possible to solve. There will always be hard problems. Single-threaded, synchronous code is not congruent with actual hardware constraints, and this has been the case for decades. And to the people advocating for async/await or green threads or similar, they are necessary, but even green threads don't compare to the ergonomics of single-threaded and synchronous code.

This is almost what we have in UNIX. The resource sharing and multithreading happens automatically if you start shelling out to programs to do work, or have a simple multi process architecture like NGINX, Postgres, and web app

The new Java runtime does this using lightweight fibres.

I mean, Swift or Rust async/await with poll or io_uring gets you there. I use Swift structured concurrency with io_uring and love it.

The only thing that's agreed upon and settled across OSs, is that reading from a regular file is always blocking, even with O_NONBLOCK.

This means that you neither epoll nor kqueue nor poll work if you ever need to read from a file.

I wish the BSDs and Linux and other platforms would align on non-blocking IO semantics. I just hope it's something new which can handle non-blocking IO with files too.

This is not the case on NT. The OVERLAPPED data structure allows for async file operations.

https://learn.microsoft.com/en-us/windows/win32/fileio/synch...

Reply on news.ycombinator.com