Hacker News Reader

Comment by clavoie | Hacker News Reader

(Disclaimer, I worked at Google over a decade ago, for about half a decade; as an SRE for about 2 of those years)

Yes, it absolutely makes sense. You never know when a typo in a config file or bug in your job-management code will go bonkers and try to take over all resources. Same think as disk quotas on computers with tons of users: you want to limit the damage that a user can accidentally (or not...) do.

Good quota systems saved my team's bacon way back when a few times, when fewer people were at the company; I can only imagine how useful they are at what, 10-20x the size?

View on HN

In particular you cannot provide an internal SLA on write requests to low-level storage solutions if you can't guarantee that you actually have the spare disks.

The SLA would have to be something like "we guarantee 99.999% write availability unless team A or team B or team C makes a mistake", so now you have to look up what team A and B and C are up to and what do they guarantee. Instead of the more sensible "we guarantee 99.999% write availability unless you are out of quota".

Exactly. At some point, someone somewhere will have a bad day / sneeze on a keyboard at just the perfectly wrong time -- at Google's scale, that's statistically a daily (hourly?) occurrence.

Quota systems for the win -- if only as a shared agreement on who can do how much damage / take how much resources before they're automatically stopped and more resources must be justified through some review process (likely involving budgetary concerns).

Shouldn't there be a soft limit that start sending out warnings before hitting the hard one ? Unless some service started eating the quota so fast that it reached both limits in quick succession, the upper limit should never be reached.

I can't comment on the current incident, as I've been gone for more than a decade.

But my recollection is that most SRE teams would have had exactly such things -- either as really crazy borgcfg (sorry, I mean kubectl) code, or as a borgmon (sorry, prometheus) alert.

The former were a real PITA to work with (borgcfg hadn't been designed with that in mind), and the latter were only reactive (and thus unable to warn about things until they were fast becoming a real problem).

So most likely those safeguards were somehow disabled or made ineffective by the exact circumstances of the problem, possibly the speed at which things developed.

From the article capacity was reduced

You're right. Still, a better approach could have been adopted, like blocking a change that would cause some service to go over quota, or have the automatic system change the quotas gradually, thus giving time for alerts and human intervention

I'm sure that this Google outage was the combination of several independent issues. Perhaps there are the kind of safeguards you mentioned in place, but they returned an incorrect view of the world because of a bug exposed by a network partition? The point is that with huge distributed systems the things take them down (at least once they're mature) are typically the result of several failures that compound and interact in a way that wasn't foreseen. IMO it's more likely that the suggestions you made (blocks and/or automation) are already in place but failed in some more complex way.

Source: worked at a FAANG for a decade, saw many big incidents. Almost all were the confluence of several smaller issues that, had they happened alone, wouldn't have been newsworthy.

Well here’s the thing though. Google are removing quotas from things like appengine. How can it be simultaneously helpful to have quotas but at the same time google wants to remove them?

I think you're conflating very different things. Quotas may be removed if you've got a virtually unlimited amount of a certain resource (and even there they can be useful to prevent overbilling), but disk quotas are necessary because disk space is not unlimited. Without those, a noisy neighbour can break your application.

Reply on news.ycombinator.com