I see what you did there :)
88 karma · joined November 5, 2016
Did you evaluate Digital Ocean as an option?
Mitigation is your top-priority. Bringing the system back to a good shape.
If there needs to be follow-up actions, take the less-impactful steps to prevent another wave.
If there was a deployment, roll-back.
My concern here is, a deployment have been made months ago, and many other changes that could make things worse were introduced. This is the case. The difference between taking an extra 10-20 minutes to make sure everything is fine, versus taking a hot call and causing another outage makes a big difference.
I'm just asking questions based on the documentation provided; I do not have more insights.
I am happy Stripe is being open about the issue, that way many the industry learns and matures regarding software-caused outages. Cloudflare's outage documentation is really good as well.
I hope that lessons are learned from this operational event, and invest towards building metrics and tooling that allows you to, first of all, prevent issues, and second, shorten the outage/mitigation times in the future.
I'm happy you guys are being open about the issue, and taking feedback from people outside your company. I definitely applaud this.
There's a 20 minute gap between investigation and "rollback". Why did they rollback if the service was back to normal? How can they decide, and document the change within 20 minutes? Are they using CMs to document changes in production? Were there enough engineers involved in the decision? Clearly all variables were not considered.
To me, this demonstrates poor Operational Excellence values. Your first goal is to mitigate the problem. Then, you need to analyze, understand, and document the root cause. Rolling-back was a poor decision, imo.
And the best thing, no assholes and backstabbing like in AWS. :)
Need an ultra sound? Wait for 3-4 weeks. Want an MRI? Wait for 2-3 months.
Why can't I get immediate care when I need it without having to step on the emergency room and pay (probably) thousand(s) USD?
2. In my case, it did not. I decided my happiness/wellness and health (physical and mental) were more important.
How long do you think it'll take him to reach his goal? He talks about starting a start-up, selling or profiting, but all of that take a lot of work. And even then there's no guarantee his plan will work.
Saving 15M will probably take a developer anywhere from 10-20 years? Or am I very wrong?
It's very nice to see Amazon wrote the article about him!
"Error establishing a database connection"