Hacker News Reader

Comment by cnnlives265 | Hacker News Reader

This is the case with just about every status page I’ve ever seen. It takes them a while to realize there’s really a problem and then to update the page. One day these things will be automated, but until then, I wouldn’t expect more of Cloudflare than any other provider.

What’s more concerning to me is that now we’ve had AWS, Azure, and CloudFlare (and CliudFlare twice) go down recently. My gut says:

1. developers and IT are using LLMs in some part of the process, which will not be 100% reliable.

2. Current culture of I have (some personal activity or problem) or we don’t have staff, AI will replace me, f-this.

3. Pandemic after effects.

4. Political climate / war / drugs; all are intermingled.

View on HN

Management doesn't like when things like this are automated. They want to "manage" the outage/production/etc numbers before letting them out.

There's no sweet spot I've found. I don't work for Cloudflare but when I did have a status indicator to maintain, you could never please everyone. Users would complain when our system was up but a dependent system was down, saying that our status indicator was a lie. "Fixing" that by marking our system as down or degraded whenever a dependent system was down led to the status indicator being not green regularly, causing us to unfairly develop a reputation as unreliable (most broken dependencies had limited blast radius). The juice no longer seemed worth the squeeze and we gave up on automated status indicators.

> "Fixing" that by marking our system as down or degraded whenever a dependent system was down led to the status indicator being not green regularly, causing us to unfairly develop a reputation as unreliable (most broken dependencies had limited blast radius).

This seems like an issue with the design of your status page. If the broken dependencies truly had a limited blast radius, that should've been able to be communicated in your indicators and statistics. If not, then the unreliable reputation was deserved, and all you did by removing the status page was hide it.

> all you did by removing the status page was hide it

True, but everyone that actually made the company work was much happier for it.

> whenever a dependent system was down led to the status indicator being not green regularly, causing us to unfairly develop a reputation as unreliable (most broken dependencies had limited blast radius)

You are responsible of your dependencies, unless they are specific integrations. Either switch to more reliable dependencies or add redundancy so that you can switch between providers when any is down.

The headline status doesn't have to be "worst of all systems". Pick a key indicator, and as long as it doesn't look like it's all green regardless of whether you're up or down, users will imagine that "green headline, red subsystems" means whatever they're observing, even if that makes the status display utterly uninterpretable from an outside perspective.

100% — will never be automated :)

Still room for someone to claim the niche of the Porsche horsepower method in outage reporting - underpromise, overdeliver.

Thing is, these things are automated... Internally.

Which makes it feel that much more special when a service provides open access to all of the infrastructure diagnostics, like e.g. https://status.ppy.sh/

Nice! Didn't know you could make a Datadog dashboard public like that!

>It takes them a while to realize there’s really a problem and then to update the page.

Not really, they're just lying. I mean yes of course they aren't oracles who discover complex problems in instant of the first failure, but naw they know when well there are problems and significantly underreport them to the extent they are are less "smoke alarms" and more "your house has burned down and the ashes are still smoldering" alarms. Incidents are intentionally underreported. It's bad enough that there ought to be legislation and civil penalties for the large providers who fail to report known issues promptly.

Those are complex and tenuous explanations for events that have occurred since long before all of your reasons came into existence.

Reply on news.ycombinator.com