Right, wrt sendfile, I remember coming to that conclusion as well. I really wanted a thread-less design, mostly for simplicity's sake and to support single core SOCs well. Is that what you do in Lwan for large files? Use a dedicated thread for socket writes with sendfile? I guess mmap + madvise could also help, but there is just no way to know for sure, so you sort of have to assume the worst?

Which ever way you do it, it's so easy to end up blocking without realizing, strace is great here to see how much time you actually spend in the kernel.

How much the aio calls block is highly filesystem-dependent. I was hoping to use btrfs for my project, but when testing aio support, I saw my demo was spending around 80% in io_submit and less than 10% in epoll_wait, where as with ext4 it was more than the opposite.

Interesting stuff, and doing it in Rust was a pleasant experience I must say, although if you are trying hard to avoid copying and allocating, the borrow checker can get a bit naggy. Worth it though IMO, and should improve in the future.