Hacker News Reader

Comment by walrus01 | Hacker News Reader

On the scale of big hosting operations, 60Gbps outbound is not that much. If you're buying full tables IP transit from major carriers at IX points, I've seen 10GbE for $700-900/mo, and 100GbE circuits for under $7k/month. Of course you wouldn't want to have just one transit, but I'm fairly sure if somebody said to me 'here's $20,000 a month to buy transit', on the west coast, it's within the realm of possible.

Ideally of course they should be able to meet a fairly wide number of downstream eyeball ISPs at the major IX points in the bay area and offload a lot of traffic with settlement-free peering.

60Gbps outbound from AWS, Azure or GCP would be astronomically incredibly expensive.

View on HN

It seems that companies can be too big for the cloud and too small for the cloud (don't need k8s). I wonder what's the sweet spot.

horizontal and vertical scaling are the latest push, but diagonal would be the sweet spot

Exactly this. There are some logistical complexities (e.g. some of our bandwidth is funded by the E-Rate Universal Service Program for libraries which runs on a July-June fiscal year and so rapid upgrades on that front aren't possible), but by and large egress bandwidth isn't our primary challenge. Intersite links as I noted in the video are the current big one, and that can and does involve occasional time-consuming construction -- but honestly over the past year, a combination of total blowout of my usual capacity planning (including equipment budgets) plus the logistical complexities of lockdown have resulted in slowness to upgrade as fast as we'd like to.

Two words: hurricane electric.

God bless HE Fremont. They are the unsung story of the Internet backbone. If one were to make a list of companies that at some point had a major fraction of their hosted physical infrastructure at HE I suspect it would make people's jaws drop.

It's been a huge blessing for a whole generation of startups to have a radically well-connected space that just about anyone can drop in equipment at multi-gig unmetered (albeit admittedly extremely constrained on power and cooling). It is honestly a part of what has made Silicon Valley great. Well, that and being able to cobble together a few replacement servers from Fry's components (RIP!) or schlep out to some ex-CTO's Sunnyvale garage in the middle of the night to offload some lightly used VA Linux 1U's...

Even today, you can get 15A and a 42U cabinet to call your own with unmetered gigabit for $400/mo - and probably less if you ask nicely.

IME with cloud in the small and in the large: network prices are artificially high on the cloud providers and are very easy to get discounted if you are a big spender.

So the small pay for the large?

Yes, that's typically how it works for any industry. You get discount on bulk orders.

It seems they are regularly maxing out their network infrastructure. If it's so cheap, how come they don't just buy more? Is it the cost of the actual hardware? (I know they recently upgraded)

They are maxing out the fiber links between their own datacenters, which is in the process of being addressed. If the bits can't get from the datacenter full of hard drives to the datacenter that connects to the internet, not much point in buying additional transit capacity.

We are upgrading again-- the pandemic has made us kind-of popular.

Because of budget we tend to use things up 100% before putting more in.

Reply on news.ycombinator.com