Achieving super low latency for critical real world internet applications (2018) | Hacker News Reader

Achieving super low latency for critical real world internet applications (2018)

netdevconf.org

81 pointsyoquan30 comments

HFT guy here.

There's problems with using the internet if you want low latency. Not just low latency, but also low jitter, ie consistent latency. You're sharing a bunch of infrastructure with other people, so sometimes your packet will have to wait. Try to ping some internet server a few times and you'll see it changes. I've done this with rented lines, and the number is the same every time.

Not inherent to the internet is the fact that REST seems to be a standard. Probably people will want a REST interface. So that means serializing and deserializing messages, and the messages are always going to be a lot longer than they need to be. It's not that you have to do it this way, though. There's also the fact that internet architectures tend to have a bunch of hops (load balancer / gateway / etc), which isn't going to help, but definitely something you could work around.

The figures he gives are interesting. If you really need under 20ms for AR to work, how is remote surgery going to work? 20ms is close to the time you get to render a frame in real time games at 60fps, seems a bit fast to me. The surgeon can't be far away if this is true.

Regarding hardware: I try to not share infrastructure. What I can not fully control can and will dysfunction, even in datacenters. For example, the LAN options are most often VLAN. So for my servers, I have extra NICs and direct connections one-to-one by crossover cable and ARP monitoring. It costs more, but it quickly pays for itself.

For software, everything is done in house. As you said, it doesn't have to be done a way or another.

I was under the impression that the most serious HFT doesn't even use Ethernet anymore. At each end, where it meets a common buy/sell platform, yes. But not for the competing HFT data link companies' inter-city messaging.

For the shortwave links in place between NY-Tokyo, for several I'm privy to details of, people have written their own custom dedicated purpose protocols that implement very low bitrate, ultra low latency data messaging with less overhead and ASIC packet processing time.

Depends on what you do.

For what I do, Ethernet is acceptable as long as the latency is below 1 ms. Here're some quick stats for what I consider good enough this specific usecase:

rtt min/avg/max/std-dev = 0.069/0.095/0.121/0.016 ms

Generally it is better to keep things simple and use existing stuff whenever possible.

If we are talking about very low latency (say below 30 us) you may need many other things first, like a good timesource.

Then indeed, when you start rolling your own stratum1 NTP servers, rolling your own protocols can be a sensible answer to the problem you have.

At a high level, what do you do with HFT? You aren't doing the trading, right? You just manage the systems behind it? So what is? Are there really bots "in production" that are making buying and selling decisions tied to real money so quickly that <1ms is needed?

The advantage is being first. If you learn in Tokyo that a stock is going to go up, you can buy all of the available stock in New York, or cancel your order to sell it. It's going to take you 50 ms to get that information to New York, but if your competitor takes 49 ms, you lose money; if he takes 51 ms, you make money. So 1 ms is important, and so is any time difference big enough to outweigh the noise in the exchange's ordering of events. I know people measuring and caring about speed gains of single-digit nanoseconds, roughly equivalent to replacing your two-metre cables with one-metre cables.

The end customer (whoever actually wants to buy stocks) doesn't need the price to update every millisecond. But no one has yet figured out a better way to run an electronic exchange than "first come, first served".

Source: also worked in HFT for a long time.

I'm not directly involved with it, but tangentially at a network engineering level. In things like the NYC-Chicago path, 1 millisecond is an ETERNITY.

The actual people working for HFT firms are probably not going to provide much insight into what's going on behind the scenes, due to very restrictive NDAs, security policies and litigious corporate overlords.

Why would you screw around with NTP when you could just use PTP capable infrastructure?

I have my reasons.

I see PTP as a good complement to NTP.

Microwave and fiber for long haul.

Ethernet is still viable in the LAN. With solarflare/mellanox nics + kernel bypass you can get a message from wire the memory in around 2-3 microseconds. The layer 3 10/40g ultra low latency switch offerings can pass a frame port to port in 200ns while layer 2 offering can do it in 5-50ns depending on how many features are stripped away.

HFT firms use FPGAs with integrated nics as well. Pci-e Latency to pass a message is around 800ns.

For the ULL crowd, when the money opportunity arises, everyone is stampeding to that order to get there first and causes a micro/nanosecond level spike of messaging/activity. FPGAs are great at staying deterministic during this burst.

But alas, i digress. Ethernet is still used and can be wicked fast.

Remote surgery is one of those 5G hype applications that nobody will really offer. What liability insurance will cover if a fiber gets cut and the resulting fail-over delay and jitter causes the robot arm to nick an important artery or nerve?

For AR/VR, you need to detect the motion, calculate the scenery, and put it on screen. This alone eats up a big part of the 20 ms budget. And to avoid motion sickness, the delay should be even shorter.

First Person View / FPV quadcopter pilots still use analog TV transmission as it has a latency of about 26 ms. As the 'dots' are sent as they are sampled by the camera, expert pilots can judge from distortions in a picture how the quadcopter has moved between two full frames. When I looked it up, the best digital FPV video systems offered about 50/60 ms delay.

Thanks. Want to point to others that might be looking at the parent comment and wondering 'what on earth...' A talk on ultra low latency, shared recently, was a great resource and expands on and adds to some of the points made above: https://news.ycombinator.com/item?id=18729822

When you start needing ultra low latency, you see many things very differently.

Take timestamping for example. NTP has unacceptable jitter.

I do low latency network architectue for HFTs; I'm not sure there's a real world application for shaving 50us off a latency for a route outside of this industry. That said I've been impressed with Cloudflares consistent global low latency while traveling. It's a habit to ping 1.1.1.1 and see if it goes above 15ms.

It's terrifying to hear him mention self-driving cars as a use case for super low latencies. If a few milliseconds of network latency compromises some safety system in a car, that's bad. I'd be engineering a car for worst case scenarios, not for reliance on a fast network. It makes me think of this. https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...

It's not clear why some folks are dredging the self-driving cars, vehicle-to-vehicle, and vehicle-to-infrastructure use cases out for 5G but when the rubber hits the road I suspect vehicles will deploy with dedicated short range communications for the bits that must be low latency and 5G for ordinary cellphone use. The wiki page has some starting places for DSRC info. https://en.wikipedia.org/wiki/Dedicated_short-range_communic... . It is terrifying.

It turns out car-to-car local networking is not really necessary. The use of 4G/5G networking + simulation based modeling and prediction gives great results. That’s what companies like Nexar [0] are doing.

[0] https://www.getnexar.com

If it requires network connectivity then by definition it’s not self-driving.

Then there are mixed reality applications. It won’t need HFT levels of low latency, but will demand a lot more bandwidth. Many of the tricks won’t work there.

That’ll require exploiting edge distribution to its limits, with an entire new level of internet infrastructure that doesn’t exist yet.

Network Next is a company working on this, not just for critical applications but also games:

https://networknext.com/concept/

If you are looking for information on low-latency computing, including some real advice how to tune for lower latency, look for info on HFT.

5ms reaction time mentioned on these slides are far cry for what computers can do. trading servers reaction time packet-in packet-out even without fpga can take as little as 20 microseconds.

Yes, but they sit right on transatlantic fibre-optic trunk cables. Presumably leased lines? It's a totally different proposition to a public mobile internet.

Many trading companies now utilize over-the-horizon radio. Its the fastest, most direct connection available. But even without this:

- cpu thermal profiles and power governors

- cpu affinity

- non-evented io(if you prefer latency over the number of clients served)

- network interrupt coalescing(actually, disabling coalescing), network msi-x, hardware offloads, 10gb fiber or DAC connections(very cheap nowadays)

- lock-free and kinda-crdt data structures(between threads)

- predictable memory allocations(better - no allocations on critical path)

- specialized logging and tracing(no stdio)

can drive you quite far.

What is over the horizon radio? I am familiar with over the horizon radar, but not radio for communications. Do you have any references?

Shortwave radio with big-ass yagis and dipoles, aimed at each other in point-to-point links. Very low data rates, carrying custom messaging protocols, but also measurably less latency on critical paths such as NY-Chicago or London-Tokyo.

There is a big crossover in tech with shortwave band military over the horizon communications systems, and also serious ham radio enthusiasts who use big yagis and antenna rotators to talk to each other from opposite sides of the planet.

Things like chained 6 and 11 GHz microwave between NY-Chicago, which was the "hot new thing" in HFT 7-8 years ago to achieve better than fiber latency on that route, seem quaint now.

Sorry, used wrong word. "Skywave" is a better term.

For VR and AR you really don't need these low latencies over the internet. As long as the visual output matches the users head movement the game assets can have noticable lag. Like most existing games, rubberbanding is an annoyance but not disorienting.

For those use cases it would be better to come up with a full 3d (not just stereoscopic) streaming video format that allows for some local camera movement.

In 5G circles there is this "vision" of VR/AR servers that reside in "edge datacenters" close to the customer. The user's movements are streamed to such a server which returns the calculated video stream. This doesn't make sense at all, of course. But you can put buzzwords like VR/AR on your shiny 5G slides..

Could RAMCloud be a candidate for this … at least to tackle the datacenter level, and —isn't the rest more or less out of control?

https://ramcloud.stanford.edu/docs/doxygen/md_README.html

A whole talk about low latency networking and not a single mention about TSN (time sensitive networking)/AVB (audio video bridging) which actually does low-latency?

Reply on news.ycombinator.com