C++20 coroutines and io_uring | Hacker News Reader

C++20 coroutines and io_uring

pabloariasal.github.io

67 pointssigna1112 comments

Hey just a heads up -- you're setting the queue depth of the ring to the number of files discovered, that's typically not the way you'd want to size the ring.

When you start to get up into higher queue depths, like 512-1,024, you benefit a lot from using flags like IORING_SETUP_SQPOLL.

If you're going to do a thread pool too, you may as well use IORING_WQ_ATTACH and put a ring on each thread that shares a single kernel poller, see the SO explanation I gave recently here:

https://stackoverflow.com/questions/73798651/how-to-create-a...

I love C++ coroutines! (The spec has a few warts and gives me enough reasons to hate them, like not being able by definition to properly inline nested coroutine calls, but I love them anyway.)

I've written several libraries for integrating C++ coroutines with stuff like io_uring, libcurl, c_ares, libpq and more. For example, this is how using my io_uring/coroutine library can be used: https://github.com/CM4all/libcommon/blob/master/test/co/RunC...

  auto result = co_await CoReadTextFile(queue, AT_FDCWD, path);
  co_await CoWrite(queue, STDOUT_FILENO, result.data(), result.size(), 0);

This opens a file, stats it, reads its contents, and writes it to stdout - all 4 I/O operations are asynchronous with io_uring. Source code for CoReadTextFile() which is also a coroutine: https://github.com/CM4all/libcommon/blob/master/src/io/uring...

Sample code for libpq: https://github.com/CM4all/libcommon/blob/master/test/co/RunC... and c_ares https://github.com/CM4all/libcommon/blob/master/test/co/RunC... and libcurl https://github.com/CM4all/libcommon/blob/master/test/curl/Ru...

I wrote all of this for proprietary applications at dayjob, but the core library is open source, as is much of my dayjob code. The I/O event loop this integrates with is also used by several open source projects I maintain, e.g. the Music Player Daemon (https://github.com/MusicPlayerDaemon/MPD/tree/master/src/eve...) which can also take advantage of io_uring, though not (yet) with coroutines, only "classic" non-blocking I/O.

My code is optimized for low-overhead; the very core doesn't even use std::function because I fear its implicit heap allocations. Long ago, I used boost::asio (which also integrates well with coroutines) but didn't like it because it was too bloated for me.

I've rarely seen other nerds talk about C++ coroutines, and never about integrating them with io_uring, made me thinking I'm the only one. But maybe all the others just don't write/blog about it - I never did either... That's why this blog was a refreshing read for me, thanks.

Others have integrated io_uring with asio, and asio with co-routines. The performance is about the same. The complexity is a matter of taste: asio offers lots built in that you would need to code yourself for the next project.

I generally lean toward lower total complexity at cost if more custom coding, but it is a big tent.

I recently built something to connect asio with gRPC: https://github.com/WolleTD/asio-grpc-callback

The channels and co_yield-capable coros from asio::experimental are quite fun to use for streams. When you have prior experience with async programming in other languages, e.g. Python or Rust, it's really handy that you don't have to define any awaitable types to use asio with coroutines.

All new c++ code I work with at bigco are c++ coroutines.

The underlying executors come from folly.

We have an eerily similar coding style... I am now unnerved.

Whoa, that exists? How about having some nerd fun and we create a PR for each other's open source project?

Interesting... mail me? I will meditate on this.

How do you deal with errors? Do you check them at every callsite and manually “bubble up”, or do you use exceptions? I’d love to use coroutines but we don’t use exceptions and I’m not super excited about handling errors manually.

> do you use exceptions?

Exceptions, sorry.

I learned C++ in the early 90ies, but didn't like it much; I needed to do plain C for a few years at dayjob and several open source projects until I was fed up with manual error handling and go back to C++. At first with a "-fno-exceptions" policy (using something similar to GLib's GError), but gave up my resistance after a few more years, and now I enjoy exceptions very much. It's not perfect, nothing is, but everything else is uglier and so much more cumbersome.

Exceptions seem inappropriate with writing non-blocking/asynchronous code, because where do you throw stuff when the caller is not on the call stack, but instead wants you to invoke a completion callback... but on the other hand, I don't want to implement two different kinds of error reporting - just look at the C++ standard library, which sometimes throws exceptions, sometimes uses std::errc - no, I wanted one single way to wrap error conditions, and I decided that std::exception_ptr is the way to go. All my error callbacks take an std::exception_ptr parameter, which they can then rethrow eventually, or pass on to the next error callback. Yes, I hate std::exception_ptr because it allocates memory on the heap, I despise implicit dynamic allocations, but everything else is uglier and so much more cumbersome. (I repeat myself.)

Error handling is dirty, no matter how you solve it, but C++ exceptions, with all their disadvantages, allow me to just consider the problem solved and go on with writing real code instead of keeping worrying everywhere. It just works.

I know many projects and corporations have a strict "-fno-exceptions" policy and will not change their minds like I did - that's a matter of personal taste.

I just converted something from libuv to asio and, while it has some advantages, it's definitely a huge dependency. And asio, perhaps in order to be standards-compliant, doesn't even do DNS asynchronously. I'd love a cleaner solution. My one painful requirement supporting both Linux and MacOS. Would you consider adding a kqueue implementation?

> Would you consider adding a kqueue implementation?

Can't do, I don't have Apple or BSD anywhere. But if code for it were submitted to me, I would gladly merge it (and try to keep the CI happy).

The MPD version of the event loop is portable and runs on macOS/BSD/Windows, but no kqueue, only poll() (and select() on Windows). This old-school API has its scalability problems, of course, but that matters not so much for MPD.

The event loop can already do Coroutines on all supported targets; here's another open source project where I use this library: https://github.com/XCSoar/XCSoar/tree/master/src/event/ - it's a flight computer (yes, for real airplanes) which also runs on Windows and macOS and iOS, and runs libcurl (and other stuff) as coroutine.

If you want to use my library and need help with integrating it or with adding kqueue support, get in touch with me.

Reply on news.ycombinator.com