Hacker News Reader

Comment by fidotron | Hacker News Reader

It's completely nuts that Firebase has this: https://status.firebase.google.com/incidents/ZcF1YDUvpdixZ2e...

"Firebase Data Connect unavailable due to a known Google Cloud global outage"

While the Google Cloud status page https://status.cloud.google.com/ says "No major incidents" and everything is green. So Google Cloud know there is an outage but just deem it not major enough to show it.

Edit to add: within 10 minutes of this post Google updated their status page. More curiously the Firebase page I linked to has been edited to remove mention of Google Cloud in the status and now says "Firebase Data Connect is currently experiencing a service disruption. Please check back for status. ".

View on HN

IIRC status pages drive customer compensation for downtime. Updating it is basically signing the check for their biggest customers, in most similar companies you need a very senior executive to approve the update

On the other side of this, Firebase probably doesn't have money at stake making the update

It is not the status page that drives customer compensation. It is downtime.

The status page is essentially an admission of guilt. It can require approval from the legal department and a high level official from the company to approve updating it and the verbiage used on the status page.

> It can require approval from the legal department and a high level official from the company to approve updating it and the verbiage used on the status page.

Is that true in this case or are you speculating? My company runs a cloud platform. Our strategy is to have outages happen as rarely as possible and to proactively offer rebates based on customer-measured downtime. I don't know why people would trust vendors that do otherwise.

I don't have any special knowledge about the companies involved in this outage. I do know most (all?) status pages for large companies have to be manually updated and not just anybody can do that. These things impact contracts, so you want to be really sure it is accurate and an actual outage (not just a monitor going off, possibly giving a false positive).

You are likely right, but it's still gross dishonesty. I'm not ready to let Google and their engineers off the hook for that.

Inter alia, "is essentially", "it can", tell us this is just free-associating.

We should probably avoid punishing them based on free-associating made by a random not-anonymous not-Googler not-Xoogler account on HN. (disclaimer: xoogler)

then it’s fucking useless. Let’s crowd source our own

That’s what Downdetector is.

https://downdetector.com/

You're in the crowdsourced version right now.

"It can", this is just free-associating, don't let it get to ya. (disclaimer: xoogler)

We tried to do that. It didn't work. Too much spam, scams, and abuse.

working on it! (valet network)

Nah, its just some client side caching / JS stuff. Clicking the big refresh button fixed it for me, 15 minutes before OP noted it.

(n.b. as much as Google in aggregate is evil, they're smart evil. You can't avoid execs approving every outage because checks without some paper trail, and execs don't want to approve every outage, you'd have to rely on too many engineers and sales people, even as ex-employees, to keep it a secret. disclaimer: xoogler)

(EDIT: for posterity, we're discussing a "overall status" thing with a huge refresh button, right above a huge table chockful of orange triangles that indicate "One or more regions affected" - even when the "overall status" was green, the table was still full of orange and visible immediately underneath. My point being, you gotta suppose a wholeeee bunch of stuff to get to the point there was ever info suppressed, much less suppressed intentionally to avoid cutting checks)

Something must be preventing them updating the status page at this point. Of course they could still deem it not enough, but just from my limited tests, docker, buf, etc (it may not be GCP that is down, but it is quite the coincidence). are outright down. I'd wager that this is much more widespread.

I'm actually on a bridge call with Google Cloud, we're a large customer -- I just learned today that their status page is not automated, instead someone actually manually updates it!

That's the case with every status page. These pages are managed by business people not engineers, because their primary purpose is to show customers that the company is meeting contractually defied SLAs.

Surelly no SLA will be based on the display of the status page...

Maybe or maybe not, but someone with nothing better to do than monitor that page out of boredom might “get on the horn” with lots of people to complain if a green check mark turns to a red X.

They aren't automatically based on that page, but seeing a red status makes it too easy for customers to point to it and go "see you were down, give us a refund".

should* be

This is actually the norm for status pages. If you look at the various status page offerings you'll see that they're designed around manual updates.

The best way to consistently having good "time to response" metrics, is to be the one deciding when an incident "actually" started happening, if at all :)

This feels very much like when facebook, locked themselves out of their datacenters. ;)

* https://www.datacenterdynamics.com/en/news/facebook-blames-m...

Except that AWS, CloudFlare and a bunch others are also down :-O

Downdetector shows they've got issues as well, but it can be fairly unreliable, as people don't know which service is behind their apps.

I at least have no issues on their services across a few regions, and their console works fine.

AWS looks ok to me?

https://health.aws.amazon.com/health/status

Perhaps CF is dependant on some GCP services?

> AWS looks ok to me?

> https://health.aws.amazon.com/health/status

Historically, the worst place to figure out if AWS is up/down is Amazons own status page.

seems like misinformation for AWS. CloudFlare probably depends on GCP.

The bigger you are, the more you want a human involved in the decision to publicly declare an incident.

Most status pages are manual.

At least some of the information has to be.

The weird part is that it took them almost an full hour to update it.

That's fairly typical. You want a human in the loop for decisions like that.

CF too: https://www.cloudflarestatus.com/

This extra funny that GCP status page even includes a “last updated” time, which is exactly built to convey possible failure to update in cases like this

No major incident as of “ Last updated time: 12 Jun 2025, 11:48 PDT”

Maybe the outage is preventing them from updating that specific page? Hmm

EDIT: Looks like it has been updated now (6:49 PM UTC)

Anytime there is an outage that affects App Engine, Google can't seem to get their status page updated for an extended period of time.

Almost an hour to update the page...

I hope this is the case, or google is super unreliable for production grade work.

:))))))

I asked testing to see if it was up, and it pointed out that Google shows nothing but Nest is showing an outage right now, lol

https://status.nest.com/posts/dashboard

Maybe their dashboard is hosted on GCP and they are displaying a cached version. :-)

More likely they are unable to update their own status page, but in either case not covering themselves in glory over at GCP right now.

AWS has this all the time. If you need to know if a service is down in a region, check for other engineers talking about it on X.

GCP just updated their status

Services are recovering in some locations it seems - Discord is healing

Status pages are PR. It gets the same PR treatment as anything else

lies, from big tech?

say it's not so!

Reply on news.ycombinator.com