Comment by neomantra | Hacker News Reader

Comment by neomantra | Hacker News Reader

Parent Full thread

Different anecdote, but similar vibe....

In ~2010, I was benchmarking Solarflare (Xilinx/AMD now) cards and their OpenOnload kernel-bypass network stack. The results showed that two well-tuned systems could communicate faster (lower latency) than two CPU sockets within the same server that had to wait for the kernel to get involved (standard network stack). It was really illuminating and I started re-architecting based on that result.

Backing out some of that history... in ~2008, we started using FPGAs to handle specific network loads (US equity market data). It was exotic and a lot of work, but it significantly benefited that use case, both because of DMA to user-land and its filtering capabilities.

At that time our network was all 1 Gigabit. Soon thereafter, exchanges started offering 10G handoffs, so we upgraded our entire infrastructure to 10G cut-through switches (Arista) and 10G NICs (Myricom). This performed much better than the 1G FPGA and dramatically improved our entire infrastructure.

We then ported our market data feed handler's to Myricom's user-space network stack, because loads were continually increasing and the trading world was continually more competitive... and again we had a narrow solution (this time in software) to a challenging problem.

Then about a year later, Solarflare and it's kernel-compatible OpenOnload arrived and we could then apply the power of kernel bypass to our entire infrastructure.

After that, the industry returned to FPGAs again with 10G PHY and tons of space to put whole strategies... although I was never involved with that next generation of trading tech.

I personally stayed with OpenOnload for all sorts of workloads, growing to use it with containerization and web stacks (Redis, Nginx). Nowadays you can use OpenOnload with XDP; again a narrow technology grows to fit broad applicability.

I discovered something similar in the early to mid 2010s: processes on different Linux systems communicated over TCP faster than processes on the same Linux system. That is, going over the network was faster than on the same machine. The reason was simple: there is a global lock per system for the localhost pseudo-device.

Processes communicating on different systems had actual parallelism, because they each had their own network device. Processes communicating on the same system were essentially serialized, because they were competing with each other for the same pseudo-device. At the time, Linux kernel developers basically said "Yeah, don't do that" when people brought up the performance problem.

I wonder if creating more pseudo-devices for in-host networking would help. Doesn't Docker do that already, for other purposes?

That makes sense, Linux has many highly performant IPC options that don't involve the network device. Just the time it takes to correctly construct an ethernet frame is not negligible.

Processes communicating over TCP via localhost should not need ethernet frames.

OpenOnload supports user-space acceleration of pipes. For our custom applications, we initiate connections over unaccelerated UNIX sockets and then "upgrade" them to accelerated pipes. It's like setting up pairs of shared-memory queues, but all operated via POSIX calls including the ability to watch them with epoll.

This a real old example of this technique using boost::asio [1]. We adapted that from the one in our own reactor library to Onload-pipe-accelerate some OSS that used ASIO.

> Just the time it takes to correctly construct an ethernet frame is not negligible.

For that reason, Onload/TCPDirect (and other vendors too) have APIs that allow you to pre-allocate and prepare packets (headers and payload), then you can do some final tweaks -- like choosing a ticker and price in an order packet -- and then send it out with minimal latency.

[1] https://github.com/neomantra/asio-pipe-transport

In fact, if I recall correctly, it doesn't seem to. I remember writing a tool that analyzed pcap dumps and initially assuming there would be Ethernet frames at the bottom, only to run into the fact that this wasn't true on lo, which had it's own frames, which I assume could vary by OS a bit.

Processes communicating via localhost should also not need TCP.

There are many applications which are network transparent and would need extra code to know how to optimize when the other process happens to be on the same machine (e.g., using Sys V IPC or Unix domain sockets). There are good reasons to optimize the case of localhost.

I am always bemused when people imagine that communication over localhost involves packets hitting an ethernet adapter.

So, what method do you propose for communicating locally which will be faster than the network ? Unix sockets ? Pipes ?

> Unix sockets ? Pipes ?

It has been many years since i tested this but unix sockets seemed slower to me than loopback TCP even

If you need real speed you should use UDP/MC, or I have used a lot of commercial middlewares which used "shared memory" transports. i.e. messaging paradigm APIs wrapping various configurable messaging middlewares to provide a fairly unified interface to the application messaging concepts.

>It has been many years since i tested this but unix sockets seemed slower to me than loopback TCP even

They're essentially the same... except they're not and there's a boatload of work in the middle that doesn't need to be done for unix domain sockets.

I can imagine some edge-cases however, where differences in stacks and configuration (in/out buffers vs. shared buffers, buffer sizes, packet loss vs. reliable, locking, blocking vs. non-blocking writes) may lead to more optimal scheduling under concurrent workloads. As soon as you have more than one thread doing something, there's a boatload of moving parts to consider.

But if you have everything properly tuned, unix domain sockets should win every time. If your results said something else, you probably weren't benchmarking what you thought you were benchmarking.

Shared memory of some kind will be fastest. It requires a re-architecture of the application, some of the times. Unfortunately, I can't find the Linux kernel mailing list response any more, but the kernel devs were not sympathetic to the use case of processes communicating over TCP on the same host.

I went down a similar path (Solarflare, Myricom, and Chelsio; Arista, BNT, and 10G Twinax) and found we could get lower latency internally by confining two OpenOnload-enabled processes that needed to communicate with each other to the same NUMA domain.

We architected our applications around that as well. The firm continued on to the FPGA path though that was after my time there.

I do still pick up SolarFlare cards off of eBay for home lab purposes.

Nice! We deploy like that too. I document how to do that with Redis here [1].

[1] https://gist.github.com/neomantra/3c9b89887d19be6fa5708bf401...

Yup, all the same techniques we used (minus Docker but with the addition of disabling Intel Turbo Boost and hyper threading).

A few years ago I met with a cloud computing company who was following similar techniques to reduce the noisy neighbor problem and increase performance consistency.

Frankly, it’s good to see that there still are bare metal performance specialists out there doing their thing.

That was such a cool comment that I actually went to http://www.neomantra.com/ and briefly considered applying for a job there and un-retiring

That reminds me of the when the openWRT and other open source guys were complaining that the home gateways of the time did not have a big enough CPU to max out the uplink (10-100mbps at the time), and instead built in hardware accelerators. What they did not know was that the hw accelerator was merely an even smaller CPU running a proprietary network stack.

That doesn't sound quite right to me. The worst CPU bottlenecks I recall for consumer routers running OpenWRT were mostly when doing traffic shaping (to compensate for bufferbloat in ISP equipment), not a complete inability to forward traffic fast enough. And when line speed really wasn't achievable with software routing, the hardware deficiency that mattered most was probably memory bandwidth rather than CPU core performance.

The kind of offload most prevalent on consumer routers is NAT performed by the Ethernet switch ASIC, which I don't think can be fairly described as an even smaller CPU running a proprietary network stack—you might have a microcontroller-class CPU core in the switch chip, but there's a separation of control and data planes.

I've worked quite extensively with openonload. There were significant numbers of "free wins" just from LD_PRELOAD though anything non trivial did require more configuration (mostly via env vars!). I've also written ef_vi applications for udp/mc pub in a similar market space to the one you describe. Whilst a little quirky in places you could write some incredibly fast stuff. on 2.xGHz xeon cpus you could send a useful datagram in 600-700ns. For a lot of applications you could even leave that in your critical path and not care.

Seems like you did something similar to what Vector Packet Processing is.

https://en.wikipedia.org/wiki/Vector_Packet_Processing

Does containerization have any impact on performance in your use cases?

I've not exhaustively characterized it, but RedHat has this comparison [1]. It is not enough for the scope/scale of my needs. I do still run bare metal workloads though.

That said, I have had operational issues in migrating to Docker, which is another sort of performance impact! I reference some of my core isolation issues in that Redis gist and this GitHub issue [2].

[1] https://www.redhat.com/cms/managed-files/201504-onload_conta... [2] https://github.com/moby/moby/issues/31086#issuecomment-30374...

Reply on news.ycombinator.com