Looking at many outages, the root cause is usually novel and the result of a combination of known and unknown changes to a system and its context. This includes your typical "operator did something too fast/too big/without code review", because there's usually something very interesting in how someone was able to do that in the first place. We should learn from them and mitigate them to our best ability, but IMO I don't think you can drive these novel events to zero.
What's more interesting (to me) is the blast radius of any given outage, along both externally visible and internally visible seam lines. For example, the EBS outage of 2011 should have been isolated to a single AZ, but caused impact in other AZs for customers because of regional coordination (and work was done to push more functionality into each AZ to improve isolation). The better we partition and isolate down workloads in our services, the smaller the magnitude of any particular incident, and the easier it is for downstream users to move around it.
With respect, less a fan of your second comment. Please know that I'm saying this with respect :)
> This is our new democracy.
I have a slight reaction to people talking about tech and "democracy". 4 people in SF can change some lines of code after a team meeting and tank a family business in mumbai. That feels to profoundly undemocratic on such a massive scale, that it hurts. (h/t recent Upstream podcast with "People's History of Silicon Valley" author, for the scenario)
Yes, we technologists sometimes feel our workplaces are more democratic compares to employers elsewhere, but outside that company, the spaces we live in our becoming less and less possible to scrutinize and speak up about.
I feel that maybe if we were running more worker co-ops in the tech industry, more ecologies in solidarity, more platform cooperatives -- then I might be able to bear using the word "democracy" to describe the things we're participating in...
> How do we convince people that we can move the world in the right direction re: pollution, human trafficking, equal rights, etc if we join up collectively?
Haven't we already shown them that we as technologists _can't_ lead that? We had a sandbox to prove something. It's San Francisco, and it's a dystopia for everyone but us. People rightfully are (and should be) very wary of trusting mainstream technologists and their worldview to solve much.
I would love to see us speak less of the power we have through occupying "structural holes" (positional power of gatekeeping a resource or skill or knowledge) and more about the power we have be being _support_ and by strengthening relationships around us. This feels important. But it also dissolves the power we know. (Lots of research suggests masculine minds ahve tactics that are more likely to seek out and occupy structural holes in social networks, whereas feminine minds wire up the network around them, you might say they "repair" the hole.)
https://en.wikipedia.org/wiki/Structural_holes
https://faculty.chicagobooth.edu/ronald.Burt/research/files/...
Anyhow, I say all this with love. I appreciate you. I just get frustrated, because I largely see the landscape of technology as breaking things and weakening the important features of our network -- features that the current cohort of technologists (through self-selection biases toward abstract thinking) perhaps can't see and don't know how to value.