Cloudflare: Your status page showed "all systems operational" for over 20 minutes while your primary domain was returning a 502 error. Please change this to update automatically, many other engineering teams depend on you.
At Ably, our status site had an incident update about Cloudflare issues being worked on (by routing away from CF) before Cloudflare did: https://status.ably.io/incidents/647
We have machine generated incidents created automatically when error rates increase beyond a certain point stating "Our automated systems have detected a fault, we've been alerted and looking at it". See https://status.ably.io/incidents/569 for example. I think much larger companies like Cloudflare and Amazon could certainly invest a bit in similar systems to make it easier for their customers to know where the problem likely lies.
Trust me, they know.
They know about problems we never find out about, too.
So our system thought none were running and so it kept launching instances....
These were SPOT instances and thus only cost like .10 per hour...
But we launched like 2500 instances which all needed to slurp down their DB and config - so it overloaded all other control plane systems...
We had to reboot the entire system. Which took forever.
The only good things was this happened at 11am - so all team members were online and avail... and then AWS refunded all costs.
---
The other fun time was when a newbie dev checked in AWS creds to git - but he created the 201th repo (we had only paid for 200) -- and as it was the next repo which wasnt paid for, it was by default public - thus slurped up by bots asap - which then used the AWS creds to launch bitcoin mining bots in every single region around the globe. Like 1700 instances.
The thing that sucked about that was it happened at like 3am and we had to rally on that one pretty fast. AWS still refunded all costs...
That's an odd choice of a failure mode.
> AWS still refunded all costs...
Yeah they should. It was their silly design choice that lead to disclosure of secrets after all.
What kind of failure mode is that even. Failing to create the repo would have led to a better user experience for sure.
Can you imagine if S3 charged more for private objects and once you reach your count, it just makes them public and posts them on Reddit?
Exactly! What a stupid design UX.
Wait, was the 200 private repos issue an AWS thing or a GitHub/GitLab/whatever thing?
What AWS product has a concept of private/public repos and limits on how many of the former you can get for a certain price?
Never post aws secrets to git.
The actual incident page was correct.
I think the only way is to have status page operated by an independent 3rd party, but I don't think there's a viable business model for someone to provide such service. Perhaps there might be even a risk of lawsuits against you.