I've spent essentially the last year trying to find the best way to use io_uring for networking inside the NVMe-oF target in SPDK. Many of my initial attempts were also slower than our heavily optimized epoll version. But now I feel like I'm getting somewhere and I'm starting to see the big gains. I plan to blog a bit about the optimal way to use it, but the key concepts seem to be:

1) create one io_uring per thread (much like you'd create one epoll grp)

2) use the provided buffer mechanism to post a pool of large buffers to an io_uring. Bonus points for the newer ring based version.

3) keep a large (128k) async multishot recv posted to every socket in your set always

4) as recvs complete, append the next "segment" of the stream to a per-socket list.

5) parse the protocol stream. As you make it through each segment, return it to the pool*

6) aggressively batch data to be sent. You can only have one outstanding at a time per socket, so make it a big vectored write. Writes are only outstanding until they're confirmed queued in your local kernel, so it is a fairly short time until you can submit more, but it's worth batching into a single larger operation.

* If you need part of the stream to live for an extended period of time, as we do for the payloads in NVMe-oF, build scatter gather lists that point into the segments of the stream and then maintain a reference counts to the segments. Return the segments to the pool when it drops to zero.

Everyone knows the best way to use epoll at this point. Few of us have really figured out io_uring. But that doesn't mean it is slower.