Hacker News Reader

Consistently low latency is great. Not just with bufferbloat but with physical latency.

A lot of city-based fiber dwellers think that just running fiber out to rural areas actually improves performance more than it does. Moving data and interconnectivity out there helps more.

They really should get out more. Or, perhaps, learn how to insert artificial, rural like delays into their networks in the city, to grok it.

(I am very busy today at the metro connect conference in ft lauderdale today, please expect sparse replies if any. If any of y'all are actually here today, I am outside with Bandwidth is a lie t-shirt on!

Oh, I absolutely believe this.

My example of Hawaii being 70ms away from Google was something that I knew about on paper but didn't truly appreciate until I was there on vacation a few years back (where the hotel wifi injected another 30-50ms on unloaded connections).

Moving interconnectivity to rural areas is tricky because most peering is where you can convince other network operators to meet you. This is a bit of a catch-22, where if you're not big enough then most network operators won't bother, and you can't get big enough unless you get these network operators on board. There's a secondary issue where just having peering isn't useful without servers (or vice versa), and you often don't have the economies of scale to justify spending millions on routing equipment, servers, etc. if you don't expect to serve many users there.

The compromise that some of these big operators (Google, Facebook, Netflix off the top of my head) make is to install cache nodes inside ISP deployments for last-mile delivery. My understanding is that this is generally a win-win for all parties involved: ISPs don't have to build out as much backbone / peering capacity, Google and co can get away with smaller deployments and piggyback off of the ISP's existing network presence, and end users have better quality of experience.

That said, for the most part, Google's cache nodes don't serve www.google.com because we generally view them as less secure, and we would have a very bad day if we had a compromise of any high-value TLS certs. (We mostly use them to serve YouTube videos, since that's where we derive the most value from bandwidth offload.)

Please explain how you get a high latency on fiber that doesn't involve bufferbloat or doing something purposefully wrong. Where are these rural delays coming from?

Especially because an extra 200 miles of fiber only increases round trip time by 3 milliseconds. How far are signals going before they get routed in the right direction?

To give a concrete example: suppose I am in rural West Virginia. I want to access HN. HN's only servers are in San Diego. My RTT to HN is limited by how my ISP can get to transit (most likely AS3356, formerly Level3, now Lumen). Let's suppose that's in Ashburn, Virginia, in Equinix's DC2 datacenter (https://www.equinix.com/data-centers/americas-colocation/uni...). (This is actually where my ISP meets Frontier en route to listserv.wvu.edu, so this isn't a totally contrived example.)

So now I have to go east 10ms in order to go 70ms west (80ms). If, say, this ISP could also reach transit in Pittsburgh, that might only be 5ms northwest followed by 60ms west (65ms).

Or if HN decided to spin up a second server location in Chicago, then I might only need to go a third of the way across the country instead of all the way (20ms or less).

This is what I believe Dave is referring to when he says that moving data and interconnectivity matters more than deploying fiber (which might save 5-10ms over cable).

For your first example, it looks like Ashburn and Pittsburgh should be less than 2 milliseconds apart? I'd expect the difference in routing to be more like 80ms versus 74ms, and that's not very much.

Having a second server can definitely have a massive impact on latency, but that doesn't sound like an ISP thing, or a rural thing, or an interconnectivity thing.

Maybe if you had fiber directly connecting Ashburn to Pittsburgh, but it's more likely that you connect Ashburn -> Philadelphia -> Pittsburgh, which is more than double the physical distance.

Just looking at distances on a map is insufficient for actually characterizing the network path.

Also: bear in mind that you need to double all these numbers when considering round trip time. I recognize I phrased it in such a way that might have been interpreted as one-way latencies, but that wasn't my intent.

Even if you had a direct path from Ashburn to Pittsburgh, speed of light through fiber would be about 3.5 ms to travel 450 miles (there and back). And while you might expect that from just plugging numbers into an equation, I have never seen anything resembling 4ms RTT between DC and New York (which are a comparable distance apart from each other) on Google's production network, even though those are definitely directly connected (6-7ms is more realistic).

> Just looking at distances on a map is insufficient for actually characterizing the network path.

I would expect big datacenters to usually have links around, and I searched those two cities in particular and there was a news article at the top about a fiber link between them.

> Also: bear in mind that you need to double all these numbers when considering round trip time. I recognize I phrased it in such a way that might have been interpreted as one-way latencies, but that wasn't my intent.

I know. I was accounting for that too.

> I have never seen anything resembling 4ms RTT between DC and New York (which are a comparable distance apart from each other) on Google's production network, even though those are definitely directly connected (6-7ms is more realistic).

How much of that is inside the datacenters? I would expect extra and slower hops for servers as compared to data bouncing from one trunk to another.

> I would expect big datacenters to usually have links around, and I searched those two cities in particular and there was a news article at the top about a fiber link between them.

That's a good point; I later looked at Lumen's network map (https://www.lumen.com/en-us/resources/network-maps.html) and saw there was a link between iad and pit. But even if you have a network link, you do need diversity. I've seen examples where an ISP decided to do maintenance in Chicago, shutting down all their peering with us in the metro; all those users we served there then transited peering in DC, where their next closest peering point was. Unsurprisingly, their users had a bad time.

> How much of that is inside the datacenters? I would expect extra and slower hops for servers as compared to data bouncing from one trunk to another.

We generally attribute less than 1ms for all the links between datacenters within the same metro area. Neither iad nor lga are exceptions to this.

I ran a traceroute just now and the the backbone hop between lga and iad was ~4.8ms. So, better than 6ms, but still not 3.6ms which you'd expect from 450 miles / (2/3*c), and definitely not the < 2ms you claim. And we're certainly not transmitting this over copper, which would get you pretty close to full speed of light but at the cost of far lower bandwidth.

This is where things stood in 2014. An update a decade later would be nice.

https://arxiv.org/pdf/1505.03449.pdf

Indeed. In practice, it looks like many things haven't actually changed, other than numbers becoming even more exaggerated. Some things I noticed in the paper:

> Undercutting a competitor’s latency by as little as 250ms is considered a competitive advantage in the industry.

I'm pretty sure my director would tell you that number today is closer to 10ms.

> While ISPs compete primarily on the basis of peak bandwidth offered, bandwidth is not the issue.

As the submission makes evident (and you are well aware), this is still very much the case today.

> For instance, c-latencies from the Eastern US to Portugal are in the 30ms vicinity, but all transatlantic connectivity hits Northern Europe, from where routes may go through the ocean or land Southward to Portugal, thus incurring significant path ‘stretch’.

Sadly, this still holds today. Almost all cables land in UK / Ireland, although MAREA does land in northern Spain, and there are a couple others in flight.

> Most routes to popular prefixes are unlikely to change at this time-scale in the Internet

Ha. Maybe at the level of ASNs, but we certainly perform traffic engineering on much smaller timescales (see https://www.cs.princeton.edu/courses/archive/fall17/cos561/p...).

Protocol improvements have definitely come a long way in the past decade. QUIC is now an IETF standard, with 0-rtt session resumption as you mention, as well as initial congestion window bootstrapping to reduce the number of round trips in slow start. But we haven't made much progress in many places that the article points out are in need of improvement.

I think the focus on speed-of-light in vacuum and the development of a c-ISP is not as useful for discussing the internet backbone, at least until we have viable replacements for fiber that are able to satisfy the same massive bandwidth requirements. Even ignoring YouTube video serving, we still have many terabits of egress globally, so the 80Gbps capacity target is not anywhere close to enough, even for 1% of our traffic in the US. That's barely enough to serve 100k qps of 100kB files. (A full page load of www.google.com with all resources clocked in around 730 kB transferred over the network, according to Chrome devtools. That's probably an argument that we should be making our home page lighter, but more than 90% of that is cached for future requests.)

And just accounting for 2x peak-to-average ratio absolutely doesn't account for inorganic demand induced by events like the Super Bowl or a popular video game release (https://www.pcgamer.com/baldurs-gate-3-launch-slams-into-ste...).

2ms was for a single direction, so 4.8 is pretty close.

Pretty close. A typical fiber latency to a CDN in-city is actually closer to 2ms. It takes 5 round trips to negotiate a TCP SSL connection (I will exclude the cost of generating the crypto), so 10ms later you start getting bursts of content. A typical cablemodem latency is 10ms, so 50ms later you start getting bursts of content. Anything over about 20ms is perceptible (human factors research, see also dan luu´s work on keyboard latency here).

So when you are talking about shaving off 6ms from a 70ms connection, it is nowhere near as dramatic as shaving off 8ms from a 10ms connection.

Now, QUIC is doing wonderful things here - 0rtt resume, and in the cloud the overall bandwidth to a given end-user is cached, but there is a really big difference between 2ms and 70! That´s just content. Geoff Huston gave a really good series of talks this week about AS paths getting a lot closer to 1 - 50 miles - across his data set, in the last 4 years: https://www.youtube.com/watch?v=gxO73fH0VqM - hopefully this will become a separate hackernews thread.

Most of my focus, however, is not on content, but communications, where we still see 300+ms glass to glass latency for videoconferencing, or worse - and bufferbloat when you are actually using the link for other things - even fiber networks tend towards 100ms worth of buffering, and fiber has a tendency to bufferbloat the wifi.... I have seen a trend in fiber towards 10ms of buffering which is actually too low and too brittle, unless fq´d and aqm´d also.

To go back to your HN example - HN loads fast because it is ONE IPv6 address (for me) and very lightweight so tcp slow start ramps up pretty darn fast, even going all the way to San Diego.

A packet capture of even a big HN discussion is really lovely, btw. More people should do that and admire it, compared to the sordid mess older websites with 100s of objects have.

> (I will exclude the cost of generating the crypto)

The cost of generating crypto is very real when you're talking about single-digit ms latencies :( RSA-2048 TLS certs add about 2-3ms to any connection, just on server-side compute, even on modern CPUs (Epyc Milan). (I believe a coworker benchmarked this after disbelieving how much compute I reported we were spending and found that it's something like 40x slower than ECDSA P256.)

> To go back to your HN example - HN loads fast because it is ONE IPv6 address (for me) and very lightweight so tcp slow start ramps up pretty darn fast, even going all the way to San Diego.

I used HN as an example not because it's bloated, but due to its singly-homed nature to illustrate how much content placement matters. Yeah, we could quibble about 80ms vs 65ms RTT from improving peering but the real win as I mentioned was in server placement. Throwing a CDN or some other reverse proxy in front of that helps as far as cacheability of your content / fanout but also for TCP termination near the users (which cuts down on those startup round trips). This is why I can even talk about Los Angeles for www.google.com serving even though we don't have any core datacenters there that host the web search backends.

(For what it's worth, I picked Chicago as a second location as "nominally good enough for the rest of the US". Could we do better? Absolutely. As you point out, major CDNs in the US have presence in most if not all of the major peering sites in the country, and either peer directly with most ISPs or meet them via IXes.)

I did slide towards mentioning QUIC there in that that first time crypto cost is gone there too. (I rather like quic), but yes, crypto costs. The web got a lot slower when it went to https, I noticed.

A highly interactive site like HN is inconvenient to cache with a CDN (presently). Also auth is a PITA. And HN, even at 70ms, is "close enough" to be quite enjoyable to use. On the other hand, most of my work has shifted into a zulip chat instance, which feels amazingly faster than any web forum ever could.

It would be cool if hackernews went more like a modern chat app.

I agree that caching is difficult in this case, but even a local reverse proxy would eliminate most of the connection setup overhead by reducing the roundtrip on ephemeral connections while keeping longer-lived connections open for data transfer (thereby also eliminating the cost of slow start, although this HN thread is only ~1 extra roundtrip assuming initial cwnd of 10 packets -- 44 kB on the wire, thanks to text being highly compressible).

TLS 1.3 also eliminated one of the round trips associated with HTTPS, so things are getting better even for TCP. And yeah, most of the time we think of the cost of crypto as the extra network round trips, which is why I pointed out that RSA is expensive (not to mention a potential DoS vector!) -- at some point your total TLS overhead starts to be dominated by multiplying numbers together.

(I like QUIC too! I appreciate that it's been explicitly designed to avoid protocol ossification that's largely stalled TCP's evolution, and so it's one of the places where we can actually get real protocol improvements on the wider internet.)

Geoff´s talk was really good. I hope discussion starts over there: https://news.ycombinator.com/item?id=39561073

If latency is consistent you get used to it and learn to deal with it. Improving latency would improve your quality of life, but you often won't realize it. I don't need the ssh to show the character I just typed instantly so long as the characters get there eventually in the right order. I don't need my bank's webpage to show my balance instantly. However if latency is not consistent I will notice as I've already learned how long those things "should" take.

> If latency is consistent you get used to it and learn to deal with it.

Too true. I played so much WoW on 56k that when I finally moved into an area that had high speed internet I basically had to learn to play all over again. IE stop compensating for pings in the ~300ms range.

Mosh, over wifi, is a sanity saver.

Having thought about this today and read some responses here, my best proposal would be to require endpoints available for the FCC to measure and then just create a service to show the results. That would give so many more vectors of improvement over time. How to measure, report such that users understand, capture change over time. No mater what number one puts on a label, that number will likely become misleading over time.