Building a database buffer pool in Zig using io_uring's new fixed-buffer mode | Hacker News Reader

Building a database buffer pool in Zig using io_uring's new fixed-buffer mode

gavinray97.github.io

158 pointsgavinray23 comments

Aha! I was thinking about trying this exact same approach recently, so I'm glad to see I wasn't insane! Very clever.

I have been "in my spare time" (that is, never finishing) building out a page buffer mgmt system based on the one in the Umbra paper (see https://db.in.tum.de/~freitag/papers/p29-neumann-cidr20.pdf) which has a kind of novel approach where they essentially allocate the same physical address at multiple virtual addresses but with different block/slab sizes so that they can reduce fragmentation and still support dynamic page sizes. Anyways, it seems this author is also on the same "page" (sorry) as me but focusing more on exploiting `io_uring`. Nifty.

I'm working in Rust, not Zig, but a lot of nice stuff to read through here.

I will have to read through this in more detail later when I have more brain-space available.

Jinx, I'm also working on this, as a tutorial series!

I had the pleasure of hearing Neumann present the Umbra architecture on the CMU DB series webinar recently.

I'm slowly working through re-implementing the Coroutines async buffer pool paper. Blocked on trying to properly implement and understand "Morsel-Driven Parallelism":

https://db.in.tum.de/~fent/papers/coroutines.pdf

I have the async, io_uring buffer pool written in Kotlin using it's native Coroutines but the scheduling and morsel stuff I'm still trying to understand

I watched the same CMU lecture, very cool.

Are you getting paid to work on this kind of thing, anywhere? Do you want to be?

  > Are you getting paid to work on this kind of thing, anywhere?

My dayjob is a fair bit higher level. I work on a sort of query engine on the JVM that translates an internal query IR/AST into dialect-specific SQL + other languages so you can query anything with the same frontend language:

https://techcrunch.com/2022/06/28/hasura-now-lets-developers...

I think it'd be neato to work on a database, but my dayjob is also pretty interesting to me.

  > I'm working in Rust, not Zig, but a lot of nice stuff to read through here.

FYI, Datadog has a Rust library for scheduling things to run thread-per-core with io_uring

It'd be really useful for DB use cases:

https://github.com/DataDog/glommio

Nifty. I started from the part of the Umbra presentation that initially struck me as most interesting -- the allocator portion (where he talks about repeatedly MMAP_ANONYMOUS on the same address range to reduce fragmentation, etc.) I stopped working when I started pondering the bits about pointer swizzling and how I'd make that work nicely in Rust, and never got as far as the pieces that would actually do the syncs to disk.

I also came across another fellow who was inspired by the optimistic locking btree presented in LeanStore Umbra: https://github.com/NeowayLabs/bplustree

One note: I mistakenly thought that fixed buffers were part of the 5.19 changes for provided buffers

Jens Axboe corrected me on this, fixed buffers have existed for a while apparently!

https://twitter.com/axboe/status/1581392910687567873?t=gx_tQ...

Aren't pages in a buffer pool usually in a different representation than the on-disk format?

Where did Zig come from?

Never did I hear about it and suddenly people talk about it.

What happened

It was first released in 2016 so it's not ultra new but it's getting more attention at least here.

If you want an overview of the language and the Zig Software Foundation:

https://www.youtube.com/watch?v=Gv2I7qTux7g

https://ziglang.org/zsf/

Portland, USA apparently (from info at https://twitter.com/andy_kelley).

Popular language of the moment. Rust, Go, Elixir. :)

Next few months will be "built X with Zig" with hundreds of upvotes.

tbh, i don't even think that promotion a product based on their language is a selling point (unless it's a lib of course!)

But this whole `built with X` is in my opinion just some 'i try to appear cool by using x language' and i don't really like it.

Don't get me wrong, i absolutely love rust, zig and go, but i believe, that you shouldn't pick a technology based on such subjective matter. A language is just a tool to solve problems, and you don't really want to screw in a screw with a hammer, do you?

Zig does look great

Reply on news.ycombinator.com