Hacker News Reader

Comment by ArtWomb | Hacker News Reader

Is the sendfile system call still the fastest way to transfer data locally on Linux systems?

For communicating between processes running locally on system. Or, for example, a server running locally on an android device. Unix domain sockets typically beat TCP connections for most large file transfers. But is there anything faster?

And what about transfers over the public net from machine to machine? I usually just use netcat, ssh, or scp.

View on HN

Consider trying rsync instead of scp - I've found it to be faster for multiple small files.

I love rsync, but being exceptionally fast is most certainly not one of it's properties.

Piping tar works much faster too. It also allows preserving metadata.

That depends on what you're doing. Sendfile, splice and copy_file_range are the fastest if you want to shovel things back and forth between file descriptors. The latter supports reflink copy if the the underlying filesystem does.

And I think there's some special casing in vmsplice that lets you transfer memory pages from process to process over a pipe. At least I've seen some patches for that, not sure if they made it into the kernel. Alternatively there's memfd.

But if you want to coordinate processes you'll use a shared memory and futexes.

If you just want to hand over an entire file to a different process you can send the file descriptor itself over a unix socket.

>>> you can send the file descriptor itself over a unix socket

Oh, that is genius ;)

Described in detail here, from Stevens Unix Network Programming

http://www.masterraghu.com/subjects/np/introduction/unix_net...

Shared memory is faster than sockets

But unsafe.

Reply on news.ycombinator.com