Hacker News Reader

The problem with readiness-based IO for disk is that without starting the request it doesn't know where it needs to read on the disk to fetch, so it can't possibly tell you if the data's ready. If you were to limit file IO to streaming rather than random-access it could potentially work, but that's not the POSIX interface. Since pipes and sockets are streams they don't have this issue.

I'm not defending the suckiness of the AIO situation on Linux, but I understand why it is to some extent.

You're right, I was working on an edit to address the problem while you wrote your comment :)

At a minimum it could tell you if the first byte at the fd offset is in cache, which it does not today. That'd at least get you a filesystem block (or more likely, 4k page). I agree this is less useful than, say, the full generality of NT IO completion.

Linux has fadvise to request an async read and preadv2(..., RWF_NOWAIT) to try to get things from the page cache. All that's missing now is a notification.

That's the key distinction between polling and select-like waiting.

> Thinking about it, readiness based reading from disk only works if you tell the system up front where to read and how much

> It would also be nice to have completion-based network I/O with AIO for symmetry reasons

I.e., you want something like NT IO completion ports, introduced in 1994.

> I.e., you want something like NT IO completion ports, introduced in 1994.

Yes, that's why I mentioned Windows.

In the intended use case, the lack of kernel-level caching can be a performance win. A database can do a better job of caching than the kernel can because it has a clearer picture of the expected access patterns. O_DIRECT in this kind of application makes a lot of sense.

A major reason to use Linux AIO is because it has no caching. That's a feature! A well-designed database engine can have a disk cache that outperforms the kernel-provided disk cache by a very large margin. Same story with I/O scheduling, which Linux AIO also lets you replace.

Almost all the performance benefits that accrue from Linux AIO for disk access is that it allows you to replace the operating system cache and scheduler with something that is better designed for your application. And large gains are demonstrably possible. If you just wanted asynchronous disk calls, there are other APIs that use thread pools for that.

A good model would have a flag that allowed databases to disable caching and everyone else to enable it, because not everything is a database engine. The absence of choice is not a feature.

At least if both choices make sense, which they do.

Sure, I wasn't clear enough — I was implying that.

You were clear enough, I was being redundant. Let's stop this here ;)

> Thinking about it, readiness based reading from disk only works if you tell the system up front where to read and how much [...], which is new API that you have to use. Reading must be done when the disk is ready and can't wait for you, so it needs another buffer and memcpy.

> So readiness-based disk I/O could work with reasonably user-friendly new API for reading and not much worse performance loss than readiness-based network I/O.

Off the cuff that new API sounds exactly like a completion-signaled read. You tell the system up from where to read and how much (the async read call), it reads when the disk is ready and buffers for you (your passed buffer and/or the cache if you're clever), and then it signals "readiness" which is the same as completion in this case.

Do you see this as distinct in any other way?

At their core, they are distinct only regarding when data is copied from kernel to user space. Readiness based copies when the application does the read() after being notified of "readiness", completion based copies (or DMAs directly from disk controller and maps into user space) before the application is notified of completion. The former is easier to use (no buffer ownership headaches), the latter is faster. Though there may be buffer management headaches in the kernel with readiness - what to do about ready reads the aren't collected? (Probably just throw them away if uncollected at the next AIO poll...)