Rethinking splice()
lwn.net
lwn.net
https://blog.cloudflare.com/sockmap-tcp-splicing-of-the-futu...
Many of the mentioned bugs are fixed by now, though arguably I'm focusing mostly on network-network splice, not disk-network transfers.
There are large benefits to be had, if one combines sendfile() with ktls:
https://news.ycombinator.com/item?id=30903146
https://people.freebsd.org/~gallatin/talks/euro2021.pdf
In traditional splice() the lack of completion mechanism was always a usability problem. Similarly with other Linux zero-copy API's. This is not a problem usually in kernel-kernel data transfers, but is when doing userspace-kernel zero-copy.
io_uring is promising.
Unix began when "networked computers" were sitting in one local network wired in a stable manner with patchcords. Then, latency of communication over the wires was no much different from writing to a local magnetic tape.
Today those latencies diverged dramatically. Your machines talk over radio, with multiple hops between cities and different providers, while local storage pushes terabytes in seconds. Trying to "splice" those two for latency gains is kinda crazy. And so are networked filesystems.
Git / Dropbox are much better approaches: you have your stuff on both ends and sync up content explicitly, without trying to provide interactive abstraction of having data locally while it actually isn't.
> It is a part of the API and it would be clearer to have it in small caps.
Very small nitpick (perhaps useful for you to be clearer in future discussion): I think you mean 'lower case letters'. Small caps are something different, and not what you'd want here: https://en.wikipedia.org/wiki/Small_caps
Am confused. Normally you'd modify buffers after they've been acked at the application layer. Do samba clients not send acknowledgements?
> the problem could be simplified by taking the pipes out of the picture and allowing one non-pipe file descriptor to be connected directly to another. The pipes, Axboe said, "do get in the way" sometimes
> a new "send file" io_uring operation could be a good solution to this problem; it could be designed from the beginning with asynchronous operation in mind and without the use of pipes
ooh I like that purely from a programmer convenience PoV
If the file is being locked against changes while the network transmission is in progress (e.g. the "stable pages" idea mentioned in the article), then in normal circumstances (unless the networking subsystem is very overloaded) the network transmission will be completed quickly, so the file will only be briefly locked against changes. By contrast, waiting for the client to acknowledge can take a lot longer, and will result in the file being locked for a lot longer, which is much more likely to have a negative impact on applications, other SMB clients, etc. Consider if you had a database file (bad practice I know, but some people will do it anyway) being read/written over SMB, blocking all writes until any outstanding reads are acknowledged could have a big negative performance impact.
Hence, they are okay with locking things in place until the network transmission is complete, but not until an acknowledgement happens. But, the splice() API doesn't provide any way to notify that the destination has finished processing all the writes the splice call sent to it.
edit: it also works poorly for blocking IO: one doesn’t splice from a file to a socket — one splices from a file to a pipe and then to a socket. Which splice() call is supposed to block?
This does not make the call useless as the performance gain is supposed to be from avoiding copying the data from the kernel to userspace which still happens.
Good luck. Splice from file to socket doesn’t send any data — it just references it. So if you mmap a file writably and then splice from it to a pipe, either splice needs to copy it or write-protect the mapping to enable copy-on-write or take a reference that will potentially propagate future changes from the mapping to the pipe. Write-protecting the mapping is slow, so pretty much all options don’t work. Blocking until the reference is gone makes no sense, because reference won’t go away after until the caller splices to the network, which will never happen. Deadlock.
But that’s not what I’m trying to convince you of. I’m trying to convince you that the semantics you seem to want cannot be implemented efficiently. The kernel isn’t a magic box that can do anything you want. It’s a computer program that happens to run at a high privilege level and gets to use some fun hardware features. There is no fun hardware feature that magically and efficiently snapshots memory without copying it. There is a magic hardware feature that “write-protects” a memory mapping and notifies the kernel when someone tries to write to it, but file data in page cache may have an arbitrarily large number of mappings, write protecting a mapping is slow (especially on x86, which is, in many respects, showing its age as an architecture), and the kind of fancy software bookkeeping needed to maintain the fiction of copy-on-write is neither straightforward nor particular efficient.
If you rent x86 servers and instruct them to fiddle with the MMU for every few kB of data sent, AWS will happily take all your money while sending a remarkably small data rate per core. You will not get Netflix-style line rate performance like this.
It’s just that Linus doesn’t want to change splice() to do what the Samba people want, which is fine but doesn’t mean it’d be impossible.
It's the other processes that make changes.
splice() connects and move data between file descriptors as if they are "pipes", but files are not pipes (in its purest sense) -- they can be seek randomly and modified concurrently by other processes.
The abstraction breaks quickly when we mix in zero-copy, caches, network packet resend, and all sorts of filesystem difference.
I think the concept remains relatively elegant in the Unix tradition (90+% value by exposing an extremely simple kernel primitive to user space), especially as compared to sendfile, but also io_uring. splice falls apart for non-blocking I/O, though. And at least sendfile has an offset parameter for block-like objects (e.g. files).
io_uring solves all of this by implementing a Go-like work scheduler in the kernel and then exposing a command queue to user space. (Both Go and io_uring use a pool of worker threads that can perform synchronous blocking task, but opportunistically make use of readiness polling or asynchronous completion for specific source/sink objects to minimize the number of active worker threads.) That's elegant in its own way, but it's totally at odds with the traditional Unix kernel/user space architecture.
Also splice supports offsets (to me your comment implied it doesn't).
Regarding non-blocking I/O, the biggest issue is splice'ing to or from a regular file I/O can still block, as file I/O in Linux is not asynchronous. For buffered I/O, io_uring gets around this by first checking if an operation would block, and if so scheduling the operation for a worker thread to perform. User space can't implement this performance hack without running into TOCTTOU issues, so you have to choose between possibly blocking, or pessimistically always pushing the operation to a thread pool.
Then there are other issues, IIRC, like the fact that readiness polling on sockets doesn't obey low water marks. Specifically, you can't set SO_SNDLOWAT on Linux; it's fixed at 1 byte. For something like splice this makes it more complex to efficiently track through polling when borrowed/gifted pages have been retired from the send queue. I think the proper method now is polling on timestamps events in the socket error queue, but this is also an indirect method. In short, it's just very difficult to track the state of pages that have been spliced, which can be problematic for splicing from a regular file (e.g. the Samba issues), but especially also for managing vmsplice buffers. In synchronous contexts, and especially simple I/O chains like a command utility implements, these aren't likely to be huge stumbling blocks; but when concurrently juggling many I/O contexts and trying to use your resources as efficiently as possible, they stick out.
I think these and other issues could be fixed or addressed, either with kernel changes or a user space library (or both), but in well over a decade nobody has; and now we have io_uring, all but guaranteeing they will never get fixed.
“Normally”, with select/poll/epoll/kqueue IO, it is entirely valid to allocate a buffer, put some data in, call send(), and immediately free it. With IOCP or io_uring or another completion mechanism, you ought to leave the buffer alone until the network stack reports completion. None of this has anything to do with SMB or other application layers — it’s how the network stack works.
With splice(), you don’t know when to discard the buffer. Even if you add application-layer acks, you are potentially vulnerable to a misbehaving peer that acks too early. For example, if you spliced from memory (using an mmapped, file-based heap, although I don’t know why one would do this):
malloc() (from mmapped space)
fill in buffer
splice to pipe then splice to socket
wait for application layer ack
free
If the ack is a lie, you have a use-after-free.