Hacker News Reader

Hacker News

Comment by ceejayoz | Hacker News Reader

Full thread

ceejayoz

Would you want your vendors publicly disclosing potentially private reasons for an outage?

View on HN

With consent, yes.

Is there any indication Railway has consented to such disclosure?

But Railway has been blaming GCP for the outage. Shouldn't GCP be given an opportunity to defend itself?

Is that what you'd want your vendors to do?

Railway hasn't placed any blame that I've seen. They've posted a timeline of what happened, without any speculation on causes.

No, that’s what the money is for.

If it were google's bug as Railway has certainly at least insinuated, then yes. I would also be fine with them saying "blah blah blah abuse was detected; we're working through it with our customer and we apologize to those impacted."

I'd also expect a story around how it is this happened w/o a human spending at least an hour working his/her way through a call list to reach someone at Railway. Starting with ops and escalating to the ceo if necessary.

yes, especially since this didn't seem to be exactly "private" where it was anything specific, it was just some kind of automated system without a human in the loop

But shouldn't that be disclosed to Railway, and not the public? If they had someone running a botnet on compromised accounts there, for example.

If Railway isn't satisfied with the explanation, they're able to say so publicly, yes?

if it wasn't something specific to their setup, it should be disclosed publicly, because this is a catastrophic incident that makes you think it could happen to you as well, and there's no way to know what could trigger it

> if it wasn't something specific to their setup

They're a web host; it could be any number of plausible mundane things that triggered automated action. This is a big recurring problem for any shared hosting provider.

a huge account like theirs should not be subject to automated actions like that.

an entire gcp project deleted along with its persistent disks.

how does that make any sense? nobody thought to call them or anything

> a huge account like theirs should not be subject to automated actions like that.

No matter how damaging the behavior?

> an entire gcp project deleted along with its persistent disks.

Railway doesn't say that - "persistent disks inaccessible", followed by "persistent disks restored to ready state". It was a suspension, not a wipe.

yes and yes, inaccessible where they had to be recovered..

there's like 0% chance there was domething super damaging going on where they couldn't get anybody on the phone yet within 10 mins or so were able to get the restoration process going with their account managers

I dont see what could be going on where an automated process would have to step in except for something like suddenly provisioning infinite amounts of resources, but quota limits should hit first so...

Reply on news.ycombinator.com