Io_submit and Linux AIO – An epoll alternative | Hacker News Reader

Io_submit and Linux AIO – An epoll alternative

blog.cloudflare.com

121 pointsmajke66 comments

The lack of clear vision in Linux system call design is a source of continual frustration. I feel like any time I or anyone else proposes generic ways to solve certain longstanding problems, the response is a hostile sort of "no" based on, ostensibly, lack of demonstrated use cases for genericity, but in actuality, I think, simple change aversion. The result is an accumulation of a pile of ad-hoc kludges and crud that do one thing or not the other (e.g., the batch-but-not-async and async-but-not-batch distinction the article mentions).

There needs to be more openness to providing general interfaces that solve whole classes of interface problems all at once, and without the technical conservatism that confronts people trying to contribute new interfaces.

I think this is based on a fear of having an API that wasn't needed and needs to be maintained forever. I bet that Linux's syscall design would be much nicer if iterating on APIs was possible (as we see in other Unix-likes).

My personally least favourite syscall design misstep is how VFS flags work. AT_ flags were supposed to be generic but we now have AT_STATX_, O_ flags are overused, and some at(2) syscalls like renameat2(2) have their own brand of flags (RENAME_). Internally VFS has uniform flags for lookup and other various things, so it's a bit silly that these flags are all basically incompatible.

(Also, the fact that O_RDWR != O_RDONLY|O_WRONLY is just silly. It takes up just as many bytes but makes life needlessly difficult when designing new extensions to openat(2).)

If I had my way, the application stable ABI would be libc, not SYSENTER. Windows gets this right and is able to iterate faster on kernel interfaces as a result.

A while ago, I proposed a kernel supplied libc add-on that would provide the syscall wrappers that glibc won't. Maybe we could declare that all applications must make new system calls through this interface and maintain syscall compatibility only for the current legacy set of system calls.

That is how it works on OS X and the BSDs as well--compatibility is maintained at the C level, not the SYSENTER level. (Although it should be noted that some of the BSDs don't guarantee forever-compatible syscall ABI in the first place).

Pros and cons to the libc ABI approach. It works very well for things that are C or mostly C-compatible (C++, maybe Rust, maybe other more esoteric languages), or interpreters with a strong grounding in C (Python; Ruby; historically, Java). It doesn't mesh well with Golang's choice to avoid linking libc, which in practice probably only works out because Linux maintains syscall ABI stability forever.

If Golang decides to avoid the platform interface library, any resulting problems are on Golang. I totally get Go's "not C" ethos, but there's no incompatibility between being not C and making system calls the way that programs are expected to make them on a platform. Go is flagrantly and needlessly incompatible with a lot of things, and there's no functionality or performance rationale for it.

There is absolutely nothing about the Go language or runtime environment that excuses them calling into libc to do platform things. Nothing. Every other managed language environment does it, and Go can too.

The big reason on Linux is libc symbol versioning. You can inadvertently link to newer versions of glibc symbols if you don’t build on a platform using the oldest libc you support. Go binaries have the advantage of being able to run on any Linux kernel they support, regardless of what distro and version they were built on.

With Rust I usually end up having to compile a bunch of build-time dependencies from source on my build images because of this, instead of just using a version of Debian etc. new enough to have the right dependencies in the package repo.

However I don’t think Go ‘s approach makes sense on some other platforms (e.g. macOS) that don’t suffer from this issue. In those cases you sacrifice forward compatibility for no real benefit.

Yep, we have an ancient Ubuntu 7.04 build server for exactly this reason - linking against an old libc.

This is how we've done it for something like 10 years now, and I've never thought about it since it works - but is there another approach we could use to link against an older libc version from a newer Ubuntu version?

Does the old libc not compile on the new system anymore? You could install it on the side, link to it, and fix up paths if needed.

With the right linker script, you too can link against whatever symbol versions you want regardless of host OS. That it's annoying to do so is a minor problem with linker interfaces, not a fundamental error in the normal program execution model.

I’m not sure where I said it was an error, let alone a “fundamental error in the normal program execution model”. It’s quite annoying either way you try to solve it; I haven’t used a linker script because then I have to hunt down each symbol version anytime I link a new symbol from glibc, instead of the future-proof setting up of an ancient build image once. If you’re writing a language forcing you to let the language compiler handle all linking of images is not a particularly user-friendly solution.

If you need to keep the platform interface library, then the kernel needs to keep supporting it. If you can't break ABI or API, then you're merely shuffling the pain point, not solving it.

The solution is to just tell people to update their code. Or accept that some bad APIs will be baked in for eternity, and deal with it.

Golang use segmented stack for goroutines. Libc support is very poor for that.

It started that way, but moved to contiguous stacks in 1.3.

> Go is flagrantly and needlessly incompatible with a lot of things

If you don't mind me asking, what are you thinking about, syscalls aside? The way they handle stack?

How about assuming TLS is at some fixed offset from an internal pthreads data structure?

Do you mean this, or was there something else along these lines?

https://github.com/golang/go/issues/23617

To be clear, I fully agree with all of that.

Go works reasonably well on openbsd despite syscall churn. You really only need to make it to uname() to decide what to do.

By contrast, Go would break constantly on macOS betas. The Mac/iOS kernel interface is libSystem, not direct syscalls.

Thankfully, they changed that in Go 1.11 (https://golang.org/doc/go1.11#runtime)

That's not really contrast; OpenBSD doesn't maintain syscall ABI stability and only supports releases for six months at a time. It's a similar problem.

By "contrast" I was referring to the statement "Go works reasonably well on openbsd." I don't consider breaking on macOS betas working "reasonably well."

I expect the statement "Go works reasonably well on OpenBSD" is being overly charitable to how frequently it breaks across releases and in reality the experience is likely similar to MacOS betas.

It also means that older compiled code might not necessarily work on new releases. This is probably not a big deal on OpenBSD, but on e.g. macOS it was big enough that they moved to libc.

Golang is the extreme exception here

I don't understand how Windows "gets this right" other than by accident. In the Win32 heyday it was in their interest to have you code against kernel32, advapi, etc. so that your stuff works on 9x and NT, maybe CE. Now 9x/CE are gone and ntdll often has the "good" interfaces and the public wrapper APIs kind of lag behind. (Maybe ntdll itself having stubs and not telling the program to SYSENTER directly is the "good" design?) Meanwhile in the Linux world you have plenty of iteration on syscalls, you might say there is too much. They have to maintain cruft but OK.

libc as the boundary doesn't make sense either as glibc is not the Linux kernel, by design, and there are other libcs in common production, like in a billion Android phones.

It's not just Windows that gets it right. The situation is the same on macOS, and on all the BSDs. The notion of userspace doing syscalls directly is really pretty much unique to Linux at this point.

And no, you still can't use NTDLL on Windows - not unless you're doing (OS-specific) drivers. It might work in practice, but it also breaks sometimes, because there's no API stability guarantee.

In those projects the libc is part of the tree. In Linux it is not the case. This is logically consistent for the project at hand. You can advocate a different way and I can agree but it wouldn't be Linux. (I am no stranger to BSD either. Been using *BSDs in some form for 19 years.)

By the way Windows CRT ships new versions with Visual Studio out of devdiv... So Windows doesn't even work that way if you draw the boundary at libc. This is part of my comment, I don't understand which part of it is MS doing it right? Ntdll abstracting sysenter? Kernel and advapi abstracting ntdll? I don't think the latter example is one of great technical execution, it just sort of happened to promote compatibility with 9x and they're stuck with it.

By "libc" I mean the userspace layer here broadly - on Windows that would be kernel32 etc. What VC++ calls CRT is built on that in turn.

Although as of Win10, the actual CRT is also an API-stable OS-level component now, not a VC++-specific thing: https://blogs.msdn.microsoft.com/vcblog/2015/03/03/introduci...

The right thing here is the general notion of abstracting away core kernel APIs in userspace. It lets you refactor the whole thing significantly without breaking everything. And userspace calls are easier to maintain long-term for legacy support reasons, as well (at some point you can make it into a DLL that doesn't even have to be distributed with the OS, and just ship that with apps that need it).

Did you miss the part where I pointed out the reason to abstract ntdll is dead? There is no more 9x. There is no more CE. It would already be serious work to move away from NT. eg. Why should all the filesystem APIs be path-based and introduce an extra CreateFile with who-knows-what sharing flags behind the scenes, when NT was built to work with handles and path lookup is slower than on Unix?

Abstractions are good but they need some basis in reality. The situation in Windows is in many cases layers atop layers without an actual need.

The difference between ntdll and kernel32 is very different from the original discussion about syscalls. The reason why kernel32 is still there because there's tons of software written for that API. The reason why it's good to abstract away syscalls is because it allows you to break those syscalls without userspace noticing.

> Windows gets this right and is able to iterate faster on kernel interfaces as a result.

The only place that'd have any bearing is for modified syscalls. Newly added obviously aren't an issue, And looking at syscalls removed, it's because they removed whole subsystems that never had anyone using it. Those would probably not even get upstreamed into the mainline kernel to being with.

I'm not seeing any evidence of what you're saying while looking at how the NT syscall layer has changed over the past couple decades.

https://j00ru.vexillium.org/syscalls/nt/64/

> The lack of clear vision in Linux system call design is a source of continual frustration.

sync_file_range and then sync_file_range2, because sync_file_range's API is literally impossible on most architectures that aren't x86.

Interesting case. However, "literally impossible" and "wastes a register padding parameters" are not the same thing.

Misremembered that, sorry.

I’m trying to convince Antirez to use this technique in redis: https://twitter.com/antirez/status/1081197002573139968

> There was a problem with that… I don’t remember what exactly, but it was like, the structures you fill did not match how Redis stored the data or something. I need to try again soon or later because the speedup in Redis would be huge.

I wonder if Linux has considered the technique that FreeBSD uses for async sendfile?

The FreeBSD sendfile() syscall is async (mostly; there is a potential to block reading metadata in VOP_BMAP). It launches a request to read disk blocks into pages, and attaches the pages to the socket buffer via mbufs, but marks them M_NOTREADY so that the protocol layer cannot send them. When the disk I/O completes, the mbufs are marked ready in the context of the disk interrupt thread, and then the proto output routine (eg, tcp_output()) is called.

This is more-or-less fire and forget for nginx. (yes, it does check the status via kqueue, but it does not block)

This was a huge performance win for us at Netflix, and the not-ready concept is the basis for our kernel-tls layer (mbufs are left not-ready, and encrypted after the disk read, and then the crypto thread calls the proto output).

async sendfile is highly specialized for ... sendfile. It'd be nice to have a more general approach in FreeBSD as well as Linux. Honestly, NT has a great model here we could adopt.

at cloudflare, we mostly don't stream large files so this is probably not a big win in our case

Hi Marek, great article as usual.

> (2) Pre-warm the disk cache with posix_fadvise(2) and hope for the best.

Just as an aside, there is also the option of mlock(2) to improve this technique as implemented in the flash webserver (https://www.usenix.org/legacy/events/usenix99/full_papers/pa...). Also some people use vmtouch (https://hoytech.com/vmtouch/) for similar things.

Regarding this new interface, the LWN article says:

> AIO poll operations always operate in the "one-shot" mode; once a poll notification has been generated, a new IOCB_CMD_POLL IOCB must be submitted for that file descriptor if further notifications are needed.

Does this mean that in the (I think fairly common) case of wanting to persist a poll indefinitely until (for example) a socket is closed, with this new interface you will need to re-issue a poll after every IO operation? If so would that mitigate some/any of the performance benefit you described?

Lastly just wanted to point out the following research in case you haven't seen it before (and the various other designs it cites):

https://www.usenix.org/legacy/event/atc11/tech/final_files/S...

Cheers!

Can anyone compare this to IO Completion Ports on Windows?

After I've read the article which kind of praises AIO, I have impression that AIO is misdesigned, have misleading name and overall quite terrible. I better stay away from it.

Also, epoll is good enough for me, my tasks. So no actual need to jump on the code to replace my epoll event loops.

Apart from the ugly API (including no glibc support), the most ridiculous part is that it only works asynchronously with XFS and without caching! Without caching! In an API that is supposed to improve performance!

I think we just need I/O readiness like epoll for disk I/O. AIO for disk seems to use the completion model ("do it and tell me when you're done" like Windows), which has higher maximum performance, but is more difficult to use / integrate. So you can now do both disk and network I/O with AIO... but it's going to be readiness based for network and completion based for disk. What the actual fuck.

Edits:

Thinking about it, readiness based reading from disk only works if you tell the system up front where to read and how much (not necessary for network sockets), which is new API that you have to use. Reading must be done when the disk is ready and can't wait for you, so it needs another buffer and memcpy. Readiness based disk writing also kind of works and it's easier - the system tells you when the write buffer has room again (threshold selectable or fixed), but submitted data is not going to be written right away. This might be the semi-reasonable explanation for the strange O_DIRECT limitation of AIO - if you need async buffered writes, just let the write cache do its work (and, you know, randomly block sometimes - never mind reading).

So readiness-based disk I/O could work with reasonably user-friendly new API for reading and not much worse performance loss than readiness-based network I/O. No idea about how much work it would take in the kernel.

It would also be nice to have completion-based network I/O with AIO for symmetry reasons and in case you really need the best performance. Though I have never worked on something where completion-based network I/O would have made a meaningful performance difference.

The problem with readiness-based IO for disk is that without starting the request it doesn't know where it needs to read on the disk to fetch, so it can't possibly tell you if the data's ready. If you were to limit file IO to streaming rather than random-access it could potentially work, but that's not the POSIX interface. Since pipes and sockets are streams they don't have this issue.

I'm not defending the suckiness of the AIO situation on Linux, but I understand why it is to some extent.

At a minimum it could tell you if the first byte at the fd offset is in cache, which it does not today. That'd at least get you a filesystem block (or more likely, 4k page). I agree this is less useful than, say, the full generality of NT IO completion.

Linux has fadvise to request an async read and preadv2(..., RWF_NOWAIT) to try to get things from the page cache. All that's missing now is a notification.

That's the key distinction between polling and select-like waiting.

You're right, I was working on an edit to address the problem while you wrote your comment :)

> Thinking about it, readiness based reading from disk only works if you tell the system up front where to read and how much

> It would also be nice to have completion-based network I/O with AIO for symmetry reasons

I.e., you want something like NT IO completion ports, introduced in 1994.

> I.e., you want something like NT IO completion ports, introduced in 1994.

Yes, that's why I mentioned Windows.

In the intended use case, the lack of kernel-level caching can be a performance win. A database can do a better job of caching than the kernel can because it has a clearer picture of the expected access patterns. O_DIRECT in this kind of application makes a lot of sense.

A major reason to use Linux AIO is because it has no caching. That's a feature! A well-designed database engine can have a disk cache that outperforms the kernel-provided disk cache by a very large margin. Same story with I/O scheduling, which Linux AIO also lets you replace.

Almost all the performance benefits that accrue from Linux AIO for disk access is that it allows you to replace the operating system cache and scheduler with something that is better designed for your application. And large gains are demonstrably possible. If you just wanted asynchronous disk calls, there are other APIs that use thread pools for that.

A good model would have a flag that allowed databases to disable caching and everyone else to enable it, because not everything is a database engine. The absence of choice is not a feature.

At least if both choices make sense, which they do.

Sure, I wasn't clear enough — I was implying that.

You were clear enough, I was being redundant. Let's stop this here ;)

> Thinking about it, readiness based reading from disk only works if you tell the system up front where to read and how much [...], which is new API that you have to use. Reading must be done when the disk is ready and can't wait for you, so it needs another buffer and memcpy.

> So readiness-based disk I/O could work with reasonably user-friendly new API for reading and not much worse performance loss than readiness-based network I/O.

Off the cuff that new API sounds exactly like a completion-signaled read. You tell the system up from where to read and how much (the async read call), it reads when the disk is ready and buffers for you (your passed buffer and/or the cache if you're clever), and then it signals "readiness" which is the same as completion in this case.

Do you see this as distinct in any other way?

At their core, they are distinct only regarding when data is copied from kernel to user space. Readiness based copies when the application does the read() after being notified of "readiness", completion based copies (or DMAs directly from disk controller and maps into user space) before the application is notified of completion. The former is easier to use (no buffer ownership headaches), the latter is faster. Though there may be buffer management headaches in the kernel with readiness - what to do about ready reads the aren't collected? (Probably just throw them away if uncollected at the next AIO poll...)

The challenge most people have with using Linux AIO is that it is an "all or nothing" proposition, you can't sprinkle a little of it onto some existing code and get a good result.

Using Linux AIO well entails writing a complete disk I/O implementation stack including your own disk cache infrastructure, I/O scheduler, etc with a high degree of technical competency and APIs to match. If you actually do all of this, Linux AIO is powerful and pleasant to use for high-performance disk access, if somewhat poorly documented. It works even better with raw block devices instead of files in filesystems -- you can pin some of its quirks on annoying filesystem behavior.

Worth mentioning you can use AIO with epoll by using io_set_eventfd.

Reply on news.ycombinator.com