Hurricane Electric 3D network map | Hacker News Reader

Hurricane Electric 3D network map

he.net

122 pointsthrowaway204835 comments

We (rsync.net) use he.net bandwidth in three of our locations - one of which is the core he.net datacenter in Fremont (which used to be an Apple Computer manufacturing facility).

The reason I originally sought out he.net to work with was their continued operation of irc.lightning.net. The MOTD on lightning referred to their network and was written in a way that made me believe they were highly technical, clueful people - I have not been disappointed.

+1.

My interactions with HE, from customer service to sales have always been excellent.

Also, they are technical. They know networking. You can use the right words, and they will understand. It is a nice change compared to other companies.

HE may be more expansive, but they are worth every extra cent.

Even their free services have great support, I had some non-standard DNS setup that wasn't possible through their web interface and it was up and running within a few hours.

I just got permission to advertise smaller subnets than a /24 during a move.

They have install times orders of magnitude less than competitors - we're talking 2 days lead time vs. 45.

And while they're not as cheap as Cogent, they are still lower cost than most other transit providers and actually have full IPv6 tables (unlike Cogent.)

I really enjoyed the IPv6 "certification" they did. If you have some free time and have never done any IPv6 networking I highly recommend it:

https://ipv6.he.net/certification/

It doesn't have the flash in the pan moment of sucesss like a unicorn startup but these guys will make billions some day due to all the peering agreements and interconnect (most the long haul is probably leased), especially as IPv6 takes majority prominence in the coming years.

Excellent slow burner kind of business and smart customer acquisition through low pricing and easy turn-ups.

Please forgive my ignorant question, but if they're already peering with a lot of networks, why would those networks change to paying for transit from them? Or do you think they'd start to steal paying transit customers away from some of the larger T1 networks?

It's possible some T1 networks may transit them for v6 as that continues to grow in usage but the largest opportunity for HE is being the highest degree v6 carrier, regardless if their v6 backbone gets labeled T1 or T2.

I don't quite know if I think they'd start raking in the billions but I think they have better long term prospects than other large scale providers at the moment.

When you peer with HE, you only get their paying customers routes. If you buy transit from HE, you get routes to the entire internet (and your routes go out to their peers).

Peering costs them little (they just have to backhaul to a customer that is paying them anyway), for transit they may have to pay Comcast, Sprint, or another provider to reach the networks they are not directly connected to.

Or in many cases they backhaul it around the world and pass it off where they have settlement-free peering. A exotic POP means little with HE unless they pick local transit too and manage to peer with eyeball networks directly.

Probably not. This is a pretty competitive space, which pushes prices towards marginal costs.

They offer service in lots of cities where there are few, if any alternatives at their price point. For instance, in western Canada, the closest alterantives at most of their POP cities charge literally 10x to 100x as much

People like you are what make these kinds of infrastructure plays such an easy turkey shoot. Look up the market cap and P/E of something like Akamai. I could build out the server footprint for that in 12 months (look up my linkedin).. the peering and lines are the hard part in terms of time and capital.

How exactly does one purchase their solution? Or more exactly what is being purchased? Do you place your servers on their rack in various data centers, and then have access to their backbone at cheap rates?

HE is a major "IP transit" provider, if you're a small to medium sized ISP, they are one of your choices in the market for buying absolutely necessary transit to reach other parts of the Internet. Grossly simplified, you as a tiny ISP get your own AS, and at least a /24 of RIPE, APNIC or ARIN IP space, set up your own router, and announce your AS and IP space to HE over a BGP session with them. Usually via a singlemode fiber crossconnect from within the same carrier hotel or datacenter.

Or if you're a colo/dedicated server company you can use them as an upstream. They also sell layer 2 transport services (built over MPLS) between any two of their POPs, for ISPs that want to establish a new POP somewhere and get 10 or 100Gbps connectivity between them.

Not a fan of LACNIC or AfriNIC?

Just using them as an example of where ISPs can obtain globally routable IP space for their own AS, also HE doesn't have so many POPs in Accra, Ghana or Panama City. Of course if you're an ISP in Ghana with a transport circuit to Telehouse Docklands, in London, and want to be an HE customer, you certainly can become one.

Thanks for the explanation. Not familiar with some of the terms but I think I get the overall gist.

Most people on this site don't in the same way they don't buy servers and prop up huge margins for AWS. The funny thing is it's not hard, I have my own personal autonomous system, IPs, and routing.

You buy transit at any "on net" site they are at, and that can include static IP routing with IPs they lease you or you use BGP to announce your own IPs (from the correct regional RIR). This is "The Internet".

Sorry, but

> Regional Regional Internet Registry

Thanks for the explanation.

There's nothing wrong with leasing transport. Almost all of NTT's network is leased, and pretty much everyone agrees they're a top flight network.

The thing that makes HE unique is that they only sell full ports. They don't do fractional billing or 95/5. You pay for 10gigs on that 10gig circuit, regardless of use. This leads to networks running HE ports near max load at peak time. From HE perspective, this is great because it's super easy to plan for and scale out. You're never surprised by a customer who goes from 3 gigs to 92 gigs on that 100gig circuit.

Never said there's anything wrong with it, it just changes the capital expenditure and worth from physical to virtual (peering agreements). Companies like Century Link, AT&T, Verizon are in a slightly different category with abundant in ground infrastructure (that has extreme tax benefits in the US).

If you look at CAIDA ASRANK, very few of the top-20 ranked ASes in the world, as ranked by diversity of peering, number of BGP adjacencies, etc also own/fully control the dark fiber their inter-city router-to-router links are operating on. They leave that to specialists like Zayo.

Telia has POPs all up and down the US west coast but it's certainly not their fiber connecting them together.

If you look globally, then owning fiber everywhere is impossible: there are countries where legal framework allows one or few specific operators.

Maybe that's location dependent? They'll sell us a lower commit than the full circuit, but we're in the SF bay area.

It's amazing to see all the underwater cabling. That's remarkable.

However, what's with the gigantic holes in interior South America, Africa, Russia, and China? Is that real or an artifact of their methodology? (Presumably, the "emptiness" of interior China is due to the fact that everything is behind the Great Firewall and is hosing their metrics)

This is just HE.net's network (with an overlay of submarine cables globally), very few if any US or EU based network operators have network presences in Russia or China (perhaps with the exception of Moscow and St Petersburg).

This is a topology diagram at layer 3 for their BGP / ipv4 and ipv6 network - nobody should confuse it with OSI layer 1 cable reality for where routes physically are, except at the most gross level of detail (POP in Paris is linked to POP in London, for example).

Somewhat related: OSI Layer 1 map: https://dev.networkatlas.org/

I've taken a look at this project in some detail, and it doesn't correspond with reality in many places. I have NDAed precision GIS datasets for things they're showing on their map, and they are nowhere near the actual location. In particular the submarine cable maps can be treated as logical diagrams and not physical reality.

The biggest carriers with fiber routes are NOT going to share the precision GIS datasets with a public site that will publish them for everyone to see. I have maps of hundreds of regen huts, vaults, aerial fiber splice cases and fiber routes for paths between seattle, salt lake city, denver, boise, sacramento and things throughout the western US, and they look nothing like Network Atlas' maps.

As a project it is an admirable goal, but I don't see the actual owners of the dark fiber choosing to make their data public any time soon.

If you look at the european POPs of major eastern european and Russian ISPs, everything in Russia that counts as a major carrier or ISP has established a presence at DE-CIX Frankfurt. The various buildings in Frankfurt that house DE-CIX switches are full of PNIs between big ISPs.

> If you look at the european POPs of major eastern european and Russian ISPs

Can confirm. (Source) When I studied in Russia, my host's father happened to be a network engineer for a big ISP in Russia.

At least for South America, the most developed cities are concentrated in coastal areas, it's where the international submarine cables land and most large datacenters are located.

The countryside is connected by the national/local telcos and thus isn't present on a international provider map. This map shows the Brazilian backbone on a macro scale: https://i.imgur.com/gRrANPJ.jpg

I still remember my networking professor showing us the HE backbone map and how blown away I was. This takes it to a whole new level!

Reply on news.ycombinator.com