Unix Domain Sockets vs Loopback TCP Sockets (2014)

Kernel bypass exists because hardware can handle more packets than the kernel can read or write, and all the tricks employed are clever workarounds (read: kinda hacks) to get the packets managed in user space.

This is kind of an orthogonal problem to IPC, and there's already a well defined interface for multiple processes to communicate without buffering through the kernel - and that's shared memory. You could employ some of the tricks (like LD_PRELOAD to hijack socket/accept/bind/send/recv) and implement it in terms of shared memory, but at that point why not just use it directly?

If speed is your concern, shared memory is always the fastest IPC. The tradeoff is that you now have to manage the messaging across that channel.

The thing that makes this possible or impossible is how your library implements the protocol, at least in C/C++. The really bad protocol libraries I've seen like for MQTT, AMQP, et. al. all insist on controlling both the connection stream and the protocol state machine and commingle all of the code for both. They often also insist on owning your main loop which is a bad practice for library authors.

A much better approach is to implement the protocol as a separate "chunk" of code with well-defined interfaces for receiving inputs and generating outputs on a stream, and with hooks for protocol configuration as-needed. This allows me to do three things that are good: * Choose how I want to do I/O with the remote end of the connection. * Write my own main loop or integrate with any third-party main loop that I want. * Test the protocol code without standing up an entire TLS connection.

I've seen a LOT of libraries that don't allow these things. Apache's QPID Proton is a big offender for me, although they were refactoring in this direction. libmosquitto provides some facilities to access the filedescriptor but otherwise tries to own the entire connection. So on and so forth.

Edit: I get how you end up there because it's the easiest way to figure out the libraries. Also, if I had spare time on my hands I would go through and work with maintainers to fix these libraries because having generic open-source protocol implementations would be really useful and would probably solve a lot of problems in the embedded space with ad-hoc messaging implementations.

If the protocol library allows you to control the connection and provides a connection-agnostic protocol implementation then you could replace a TLS connection over TCP local sockets from OpenSSL with SPI transfers or CAN transfers to another device if you really wanted to. Or Unix Domain Sockets, because you own the file descriptor and you manage the transfers yourself.

This drove me nuts for a long time, trying to hunt down why the socket couldn't be created. it's a really subtle limitation, and there's not a good error message or anything.

In my use case, it was for testing the server creating the socket, and each test would create it's own temp dir to house the socket file and various other resources.

> In general, I choose a shared memory queue for local-only inter-process communication.

Do you mean the sysv message queues, or some user space system? I've never actually seen sysv queues in the wild, so I'm curious to hear more.

https://news.ycombinator.com/item?id=24968260 Talking about using kernel bypass on pipes accepted over a UNIX socket. Link to an old asio example implementation on GitHub

https://news.ycombinator.com/item?id=31922762 Kernel bypass to FPGA journey, followed up with some user-space pipe talk with others

I do tend to use accelerated TCP loopback instead of the UNIX pipes, was just easier operationally across a cluster to use TCP.

How do you have your file delivery, database, and business logic "talk" to each other? Everything on the same computer is a "pretend network" to some extent, right? Do you always architect your own database right into your business logic along with a web-server as a single monolith? One off SPAs must take 2-3 months!

Huh, I would think UNIX sockets would be easier; since sharing the socket between the host and a container (or between containers) is as simple as mounting a volume in the container and setting permissions on the socket appropriately.

Using TCP means dealing with iptables and seems... less fun. I easily run into cases where the host's iptables firewall interferes with what Docker wants to do with iptables such that it takes hours just to get simple things working properly.

If you are using vsocks between two VMs as intended then they have the advantage that they allow communication without involving the network stack. This is used by VMs to implement guest agent communications (screen resizing, copy and paste and so on) where the comms don't require the network to have been set up at all or be routable to the host.

While Server 2003 is getting more rare and the last sighting of Windows 98/2000 has been a while, they're all running at the very least a few months after the last free security support is gone. But whether that's something you want to support as a developer is your choice to make.