Importing 3m rows/SEC with io_uring

> But utilizing these hardware capabilities with traditional blocking interfaces, like pread(), would involve many threads and, hence, some overhead here and there (like increased memory footprint or context switching). Moreover, QuestDB's threading model operates on a fixed-size thread pool and doesn't assume running more threads than the available CPU cores.

> Here we're using 8 threads to make blocking read calls to the same CSV file and observe 223MB/s read rate which is not bad at all.

Why is this? This is not an IO efficient usage of threads. And using io_uring to circumvent that bottleneck might be a solution, but just bumping the threadcount for IO seems a lot more straightforward. Using one thread per CPU core is a decent baseline if everything is async, but for blocking IO you would want a lot more threads.

Thank you for sharing the post! I’m vlad, cto of questdb - we went all the way trying to optimize CSV ingestion with io_uring, and ended up benchmarking against some OLAP peers in the process, Any feedback is greatly appreciated !

Don't assuming io_uring is always faster or much faster. Sometimes it is. But last time I benchmarked io_uring, it was good enough, but I was able to get measurably faster from syscalls and a well-crafted userspace thread pool for storage to NVMe.

(Note: the I/O thread pool I wrote came out noticably faster than fio, the benchmark tool, so don't assume fio results are the best possible.)

For that reason, I have a small application which can be configured for either io_uring or the thread pool, and the thread pool is preferred for performance.

> Here we're using 8 threads to make blocking read calls to the same CSV file and observe 223MB/s read rate which is not bad at all.

Matthias247 (sibling comment) is right on about the number of threads. Blocking I/O threads should be more numerous than cores for performance, after all they spend part of their time not running. Unfortunately it's tricky to get the balance right or to auto-tune it: They should be more numerous for I/O operations that block, but not more numerous for I/O operations that go to the page cache without blocking.

223 MB/s read rate is not fast on a modern NVMe; that's about the speed you may get from a modern HDD if not seeking. I get up to about ~4GB/s reads per NVMe (a Samsung SSD from last year in a cheapish rented server), and in a 3xRAID that speed is multiplied up to ~12 GB/s. That's with threads, not io_uring, doing small reads (4kiB) random access at ~3000 kIOPS. This is for a small, fast database engine of my own design.