Standing on Distributed Shoulders of Giants | Hacker News Reader

Standing on Distributed Shoulders of Giants

queue.acm.org

78 pointsyarapavan17 comments

If you're interested in this, the first relationship between Distributed Systems and Relativity was realized by Leslie Lamport. Read about it on his webpage, which includes his thoughts on the paper, as well as a link to the paper:

http://research.microsoft.com/en-us/um/people/lamport/pubs/p...

"Isaac Newton (1642 - 1727) was a brilliant physicist who defined the foundations for classical mechanics, laws of motion, and universal gravitation. He also built the first refracting telescope, developed a theory of color, and much more."

Newton invented the reflecting telescope, the refracting telescope was invented around 1600.

"Latency is not decreasing too much and is limited by the speed of light."

This is not true in any realistic distributed system. Sure latency is not decreasing much, but it is several orders of magnitude higher than the speed of light. There's a lot of fat to trim if we are willing to try.

In context, I'm pretty sure it was just saying "Unlike computation, bandwidth, and memory size, we haven't seen much improvement in latency, and even if we focused on it, we have a very clear limit we can't get past".

I.e., latency is -always- going to be an issue, for everyone doing anything distributed; it will never go away, or become so small as to effectively be ignorable. You can trim the fat, which can improve the real-world experience of certain things, but it doesn't actually solve, or even simplify, the hard edge cases. It just may make them rarer.

The idea that latency is currently bounded by the speed of light is what stops people from fixing the fact that it is not. This is my objection.

Stating the problem might provoke people to work on fixing it?

What is the upper bound on latency?

Latency from SF to NYC is going to be around 10ms if you tunneled straight through the earth at the speed of light. There are a ton of "realistic distributed system[s]" that are not several orders of magnitude higher than that.

I don't have servers in SF and NYC to play with, but I'd guess that if you ping between them, you'd see much higher latency than that.

Actually I do have servers in London and Sydney, so here goes.

Speed of light is 299792458 m/s (according to google)

Distance between London and Sydney is 16,983Km (according to google)

So, 16,983 / 299792.458 * 1000 = 56ms.

Now:

PING xyz.com (220.233.xxx.xxx): 56 data bytes

64 bytes from 220.233.xxx.xxx: icmp_seq=0 ttl=46 time=409.263 ms

64 bytes from 220.233.xxx.xxx: icmp_seq=1 ttl=46 time=432.955 ms

64 bytes from 220.233.xxx.xxx: icmp_seq=2 ttl=46 time=351.018 ms

64 bytes from 220.233.xxx.xxx: icmp_seq=3 ttl=46 time=373.894 ms

64 bytes from 220.233.xxx.xxx: icmp_seq=4 ttl=46 time=392.918 ms

64 bytes from 220.233.xxx.xxx: icmp_seq=5 ttl=46 time=412.357 ms

Pretty much 10x the speed of light. So, not several orders of magnitude, but at least one. This gets way worse in datacetner networks.

SF -> NY latency is about 7x speed of light[1]

https://ipnetwork.bgtmo.ip.att.net/pws/network_delay.html

The speed of light in a glass fibre is 2x10^8 m/s (according to Google). However the path through a fibre is not straigt. According to physics.stackexchange.com (http://physics.stackexchange.com/questions/80043/how-fast-do...), the speed of light though a fibre would be equivalent to 1.42x10^8 m/s.

Then you've got to account for the round-trip. You've only covered half the distance of the ping.

You also have to account for a non-direct path from London to Sydney.

All of this means you're looking at about 200ms or more as the lower bound, so your 10x becomes < 2x.

Fair point about only taking half the ping distance.

On the other issues, I disagree. They are the precisely the point of my objection. Assuming that latency is governed by the speed of light fails to take into account all of those other effects. Many of which can be mitigated. And, this is an extreme case. London to Sydney is probably one of the longest paths you can take (except maybe London Auckland?) with many stretches of fibre. On shorter paths, the overheads stack up much more heavily. In datacenters, the time of flight is a few nano-seconds, but the overall latency is tens of micro-seconds to tens of milli-seconds.

Isn't it odd that we accept 2x-5x (sometimes even 1000x) overhead over the best case and yet still argue that it's governed by the speed of light? How many other places in computing are we willing to accept that?

Another set of masterful distributed system engineers you can look at are the US founding fathers. Like any implementation, it's far from perfect, but I marvel at a system of government with enough fault tolerance, failover, and backups built in to survive as long as it has.

IIRC it began as a master-slave architecture.

Given our current computer architecture models, we'll run into Planck's constant or some other universal constant eventually. Already the path forward has more to do with increasing the number of cores rather than the performance of a single core.

"In a distributed system, you can know where the work is done or you can know when the work is done but you can't know both."

A simple thought experiment will quickly tell you that's not true. Given a task queue, the compute process that pulled the task and put a result could easily provide where and when and give you both.

Reply on news.ycombinator.com