Hacker News Reader

Comment by jordanthoms | Hacker News Reader

Google often has a outage or two around this time of the year when all the US schools come back and millions of students log in at the same time.

View on HN

I thought that's because this is the time when interns' code get deployed

Sounds pretty anecdotal. Can you back up this claim?

We run a app making millions of API calls to Google every day, so they show up in our monitoring, even if they don't make the status page. It's a pattern I've noticed from getting paged at 3am (NZ time) this time of year going back the last 4 years, I don't have hard data at hand for it though. It's not the same thing every year either - this time OAuth got hit hard, previously we've mostly seen slowdowns and higher error rates on the Drive APIs.

Says something that they know the what, where, when, and why, but the still don't (or won't) prevent it.

It's a matter of cost.

It's a temporary spike of traffic far outside of average use.

In order to handle it, they'd need a lot of resources that just aren't justifiable for the rest of the year.

You seem similar things when places sell highly desired items in online stores.

They're a bleeding cloud provider. I think they can temporarily scale up a bit.

Temporarily build a data center?

Temporarily shut down other people's services to favor their own?

I guess you're not familiar with how cloud services work. There's (generally) plenty of excess capacity.

It's the (generally) that's the issue here, now isn't it?

I didn't see any notifications that GCP was out of capacity. Did I miss one?

Gcp capacity and capacity for Google services is fairly isolated. They never coexist on the same racks

Doubtful. I put that in since absolute statements are always dangerous. I'd say at any given time, there's an excess of 20-50%. There's virtually no chance they were lacking capacity.

Within Google, there is plenty of wasted capacity, but getting capacity for a service to scale up is a fiendishly difficult task. They even employ hundreds of people whose main role is to try to allocate all the various types of resources to teams. "Oh - you want 1700 GB of bigtable in Reliability zone 3 in Atlanta? I'm afraid your department doesn't own any there - you can trade with the ads department who aren't using theirs, if you give them 2Tbits of network bandwidth between Peru and Brazil? That trade will only be till the end of the quarter though, because then they need it back."

Reply on news.ycombinator.com