Hacker News Reader

Comment by adontz | Hacker News Reader

After I've read the article which kind of praises AIO, I have impression that AIO is misdesigned, have misleading name and overall quite terrible. I better stay away from it.

Also, epoll is good enough for me, my tasks. So no actual need to jump on the code to replace my epoll event loops.

View on HN

Apart from the ugly API (including no glibc support), the most ridiculous part is that it only works asynchronously with XFS and without caching! Without caching! In an API that is supposed to improve performance!

I think we just need I/O readiness like epoll for disk I/O. AIO for disk seems to use the completion model ("do it and tell me when you're done" like Windows), which has higher maximum performance, but is more difficult to use / integrate. So you can now do both disk and network I/O with AIO... but it's going to be readiness based for network and completion based for disk. What the actual fuck.

Edits:

Thinking about it, readiness based reading from disk only works if you tell the system up front where to read and how much (not necessary for network sockets), which is new API that you have to use. Reading must be done when the disk is ready and can't wait for you, so it needs another buffer and memcpy. Readiness based disk writing also kind of works and it's easier - the system tells you when the write buffer has room again (threshold selectable or fixed), but submitted data is not going to be written right away. This might be the semi-reasonable explanation for the strange O_DIRECT limitation of AIO - if you need async buffered writes, just let the write cache do its work (and, you know, randomly block sometimes - never mind reading).

So readiness-based disk I/O could work with reasonably user-friendly new API for reading and not much worse performance loss than readiness-based network I/O. No idea about how much work it would take in the kernel.

It would also be nice to have completion-based network I/O with AIO for symmetry reasons and in case you really need the best performance. Though I have never worked on something where completion-based network I/O would have made a meaningful performance difference.

The problem with readiness-based IO for disk is that without starting the request it doesn't know where it needs to read on the disk to fetch, so it can't possibly tell you if the data's ready. If you were to limit file IO to streaming rather than random-access it could potentially work, but that's not the POSIX interface. Since pipes and sockets are streams they don't have this issue.

I'm not defending the suckiness of the AIO situation on Linux, but I understand why it is to some extent.

At a minimum it could tell you if the first byte at the fd offset is in cache, which it does not today. That'd at least get you a filesystem block (or more likely, 4k page). I agree this is less useful than, say, the full generality of NT IO completion.

Linux has fadvise to request an async read and preadv2(..., RWF_NOWAIT) to try to get things from the page cache. All that's missing now is a notification.

That's the key distinction between polling and select-like waiting.

You're right, I was working on an edit to address the problem while you wrote your comment :)

> Thinking about it, readiness based reading from disk only works if you tell the system up front where to read and how much

> It would also be nice to have completion-based network I/O with AIO for symmetry reasons

I.e., you want something like NT IO completion ports, introduced in 1994.

> I.e., you want something like NT IO completion ports, introduced in 1994.

Yes, that's why I mentioned Windows.

In the intended use case, the lack of kernel-level caching can be a performance win. A database can do a better job of caching than the kernel can because it has a clearer picture of the expected access patterns. O_DIRECT in this kind of application makes a lot of sense.

A major reason to use Linux AIO is because it has no caching. That's a feature! A well-designed database engine can have a disk cache that outperforms the kernel-provided disk cache by a very large margin. Same story with I/O scheduling, which Linux AIO also lets you replace.

Almost all the performance benefits that accrue from Linux AIO for disk access is that it allows you to replace the operating system cache and scheduler with something that is better designed for your application. And large gains are demonstrably possible. If you just wanted asynchronous disk calls, there are other APIs that use thread pools for that.

A good model would have a flag that allowed databases to disable caching and everyone else to enable it, because not everything is a database engine. The absence of choice is not a feature.

At least if both choices make sense, which they do.

Sure, I wasn't clear enough — I was implying that.

You were clear enough, I was being redundant. Let's stop this here ;)

> Thinking about it, readiness based reading from disk only works if you tell the system up front where to read and how much [...], which is new API that you have to use. Reading must be done when the disk is ready and can't wait for you, so it needs another buffer and memcpy.

> So readiness-based disk I/O could work with reasonably user-friendly new API for reading and not much worse performance loss than readiness-based network I/O.

Off the cuff that new API sounds exactly like a completion-signaled read. You tell the system up from where to read and how much (the async read call), it reads when the disk is ready and buffers for you (your passed buffer and/or the cache if you're clever), and then it signals "readiness" which is the same as completion in this case.

Do you see this as distinct in any other way?

At their core, they are distinct only regarding when data is copied from kernel to user space. Readiness based copies when the application does the read() after being notified of "readiness", completion based copies (or DMAs directly from disk controller and maps into user space) before the application is notified of completion. The former is easier to use (no buffer ownership headaches), the latter is faster. Though there may be buffer management headaches in the kernel with readiness - what to do about ready reads the aren't collected? (Probably just throw them away if uncollected at the next AIO poll...)

The challenge most people have with using Linux AIO is that it is an "all or nothing" proposition, you can't sprinkle a little of it onto some existing code and get a good result.

Using Linux AIO well entails writing a complete disk I/O implementation stack including your own disk cache infrastructure, I/O scheduler, etc with a high degree of technical competency and APIs to match. If you actually do all of this, Linux AIO is powerful and pleasant to use for high-performance disk access, if somewhat poorly documented. It works even better with raw block devices instead of files in filesystems -- you can pin some of its quirks on annoying filesystem behavior.

Worth mentioning you can use AIO with epoll by using io_set_eventfd.

Reply on news.ycombinator.com