Stripe was down | Hacker News Reader

Stripe was down

status.stripe.com

90 points1as20 comments

I work at Stripe. We're back up now. Your customers can safely retry any failed payments. We'll post a public postmortem over the next few days with more information.

If you need any help, please drop us a line at support@stripe.com (and feel free to CC me; I'm michael@stripe.com).

thanks for the responsiveness.

"Elevated error rates" is now status page speak for "shit is totally down". Amazon does the same thing with a small note next to a green check when their api is also broken. I wonder what hellscape on earth would be necessary for amazon to drop to a red icon.

It happened during the DynamoDB outages but only for some subservices.

What I'm about to say is definitely NOT IN ANY SENSE Schadenfreude, but...

I get really excited when amazing services we've come to take for granted goes down. I'd put places like Amazon, Netflix, Gmail, etc in this category. The reason? Well, I really really look forward to reading the post-mortems and getting a sneak-peek at the inner workings of said services.

Oh great, I thought I was the only one!

Don't be so excited! Maybe they just dropped another DB index and the guy who did it still has no visibility into the actual DB load.

https://support.stripe.com/questions/outage-postmortem-2015-...

At the time of writing, this post is 11 minutes old, but the status page says they have been down for 5.

Hmm!

I think generally when every alarm bell in your monitoring system goes off the first thing you do is question whether monitoring is broken. When you confirm there is a problem this big, you panic and try to fix it really fast. Then you call your other on-call guys and tell them you actually have an "oh, shit" situation.

Once you recognize there's a serious problem, THEN you make the public announcement. Ah, the life of ops.

I tried ordering from doordash about 30 minutes ago (approximately 7:20pm PST) and my card got declined, which was presumably because of this outage.

They also don't seem to count the "partial degradation" periods as downtime; Stripe.js and Checkout.js both have a 9m red bar, but Checkout.js has had a lot of "partial degradation" periods. Both sit on 99.992% uptime, though. Maybe that's not so bad; it would depend on how severe a "partial degradation" can be and still be classified that way, I suppose.

For payment processing, I've built in a system which uses stripe (credit cards only) and bitcoin (using a different API. But that is only to detect whether payment has received. If the API is down, the detection of payments can still be done manually thanks to the fact that the bitcoin network is decentralized).

I never rely on a single point of failure. And of course, always good to have a "plan B" in place.

all services are back online

Grinch's XMAS DDOS blackmail?

No better time than Christmas to have your payment platform go down :(. I can't imagine the clusterfuck at the Stripe offices atm. Overnighters ahoy!

Reply on news.ycombinator.com