> ... We don't use this anymore, in part because the number would be going down and we can't make it go up anymore ...
https://youtu.be/ZSRHeXYDLko?t=1704
I don't have a strong opinion of an hour of downtime is bad or not, and I think the concern about five nines for everything was silly. But it's an interesting talk I think it's worth watching. Is software getting worse and could we reverse the trend if we tried?
However, he has also not worked in a tech company of the kinds we see today (games are a different business), and he hugely underestimates the necessary complexity of systems. He is right that there's a lot of unnecessary complexity, and that we should push back on complexity of all kinds, but I've always found his opinions around this to be naive, to lack nuance, and to lack experience. His arguments often boil down to criticising progress rather than acknowledging the issues progress brings and finding ways to work around or minimise those issues.
Getting to five 9s is hard, and as software does more it'll get harder, but it's not impossible, particularly if we look at five-9s from the user's perspective, as clients get smarter we have more leeway to accept minor downtime without harming the user experience.
I think part of the problem is that we are focusing on the wrong thing in a lot of areas. Complexity is the real evil most of us suffer from.
Achieving 9 9's is not something you can explicitly set out and do as your primary objective. It is a side-effect of other quality choices made in aggregate.
Looking at a subset of an entire system and asking "is that database resilient to x 9's" and then worrying about that as your new target is how you wind up with something so inherently complex that the effective resiliency is somewhere around a single machine circa 1999.
Instead of looking at every piece of hardware as a potential time bomb, you could start to embrace higher-quality components and arrive back at something that resembles a mainframe again.
Complexity comes from hiring. Nobody wants to work on old reliable tech because it’s boring and uncommon.