Hacker News Reader

Comment by kazinator | Hacker News Reader

I did something very similar for a logging system, more than ten years ago.

Circular buffers were shared between user space and kernel. (Allocated by user space, mapped into kernel with get_user_pages.)

User space threads just added messages to their circular buffer by copying in data, hitting a barrier and updating a head pointer.

Like io_uring, I avoided too many system calls by having a kernel thread poll for changes. User space would only make a system call when the ring would near a "high water mark", like 85% full or whatever.

The circular buffers could outlive the process which created them and attached them. A process could crash and go away; the kernel thread would still process the last messages written right up to the crash.

Not everyone around was keen on the idea. What do you mean just stick it into a buffer and the kernel will pick it up. Polling? Bah.

View on HN

Interesting implementation. How did the buffers survive process crash? Is that a side-effect of get_user_pages or did you have a userspace daemon?

It was a side effect of get_user_pages, which obtains durable references to pages that have to be explicitly dropped. These were held by the logging module independently of the processes interacting with it.

(The kernel thread was actually provide by a system call from user space. So there was a daemon as a user space program, but that just called into the kernel, where it then sat in a loop. This is nice: you have something in user space that you can kill with SIGTERM or whatever and restart by running some /usr/bin/executable.)

If the daemon terminated, that didn't destroy any of the buffers; they would just sit there getting backlogged. New ones could be created, too (via ioctl calls on a character device in /dev).

One aspect was that get_user_pages doesn't give you a linear block of memory. I wrote the grotty C code to dequeue messages from the non-linearly-mapped circular buffer. The message headers could straddle page boundaries, so I copied those out into temporaries before working with them. I think there was some attempt to avoid making unnecessary copies of the bulk data.

> One aspect was that get_user_pages doesn't give you a linear block of memory. I wrote the grotty C code to dequeue messages from the non-linearly-mapped circular buffer. The message headers could straddle page boundaries, so I copied those out into temporaries before working with them. I think there was some attempt to avoid making unnecessary copies of the bulk data.

Open question: would it have helped to have a CMA[1] region managed by a kernel module and mapping that memory/regions of that memory to user-space using remap_pfn_range[2] instead?

[1] CMA documentation file: https://lwn.net/Articles/396707/

[2] https://www.kernel.org/doc/htmldocs/kernel-api/API-remap-pfn...

That jogs my memory; in fact, aha, I used get_user_pages in an early version, and switched to remap_pfn_range; then I nuked all that complicated non-linear access code. The ioctl for attaching the buffer then allocated the memory, and its interface changed to passing down a size and receiving a pointer.

Reply on news.ycombinator.com