This feels a lot more like the latter, and it's wonderful to see. I'm sure it's a lot harder to write, but it shows a heck of a lot more integrity.
For instance, losing three backup generators in ten minutes (as happened to someone else a while back) might be bad luck, but I'd want to see the maintenance logs and talk to the vendor. Because what's much more likely is that they hadn't been maintained properly, or there was an manufacturing defect in that run and they all came from the same batch (see also, clustered hard drive failures).
If their upstreams have had intermittent problems before, it was only a matter of time until they both had problems at the same time. And capacity planning is one of those strategic decisions that gets bland on the engineering team, who never get what they need because they're a 'cost center', which is a very different and highly damaging way of thinking about the cost of doing business (intrinsic costs).
> If their upstreams have had intermittent problems before
Sorry, this might not have been clear. We had never had major issues with any of our upstream providers before -- beyond scheduled maintenances there had never been a service interruption over a number of years.
There's almost always the risk of coordinated failure, no matter how many levels of redundancy you put into place. The cost gap between 99.99% and 100% is humongous for this reason.
From these failures, we learned some valuable lessons. One of our transit providers was acquired a year or so back, and it's apparent that things have dramatically changed in how they operate their service and respond to issues. We won't be continuing the customer relationship there for much longer unless we see positive steps taken. Another transit provider has been much more responsive and the conversations have been more productive -- we can work through the issue we observed and get back to a positive situation there (I think).
And then finally, we need more diversity. We turned up another circuit yesterday, and I'm still working on adding another 1-2 providers in addition to more peering or direct connect situations. These take a bit of time, but it's work we need to do before we actually require it, because by then it's too late. We're working as quickly as possible to get this part of our infrastructure on rock solid footing.
> capacity planning is one of those strategic decisions that gets bland on the engineering team
Absolutely. We were doing capacity planning, but it wasn't anyone's idea of a good time.
We are correcting this with the resources we have today, and we're going to do a better job in the short term. In the long term, this is a non-negotiable core aspect of someone's job -- either already on the team, or a new role that we create. It can't be a secondary, rainy day kind of task you take on time permitting and it also can't be a task that we're not well equipped to handle on a regular basis.
The driving reason that led us to start imgix, and then spend years of our lives growing and operating it, is to empower our customers to build awesome stuff on the Internet. We have to work hard to keep up our end of the bargin, and we also have to forge our partnerships and relationships with customers in a million little ways. I hope that through adversity we've forged some new relationships with customers in the way we've communicated this time (and how we will continue to communicate).
As an aside, I was personally impressed by how one of our vendors handled a service issue they had last year [1]. That was a definite inspiration for how we handled this communication.
[1] https://ns1.com/blog/how-we-responded-to-last-weeks-major-mu...