"Finally, we are in planning to remove Google Cloud services from our data plane’s hot path, and keeping them only for secondary/failover."
That's pretty clear. Google can no longer be trusted as a B2B service provider.
That's pretty clear. Google can no longer be trusted as a B2B service provider.
And as we know from the recent Gemini ban wave, you can get suspended just because.
Nearly all these linkages are due to people sharing recovery email addresses and phone numbers. Don't do that.
Adults have multiple emails so they won’t have to share it.
If something takes out the family email account, that’s fine. The only thing going there regularly are school notices, contractor receipts and recovery emails.
Google has always acted as if they have no obligations whatsoever to their paying customers.
Maybe I'm getting old but here[1] is a HN comment from 17 years ago complaining about Google banning accounts "by mistake" and having no recourse but to post on HN and hope Matt Cutts sees it and helps, and saying "there are literally 1000s of such stories for many years all over the blogoshphere and forums" which is something I remember from HN of years ago.
Which this company didn't. They managed 125b of assets belonging to other people, they didn't have 125b of their own.
They have 125b they can literally write a check against and allocate any way they want as long as it delivers an adequate return and doesn't piss off the shareholders. Other than sophistry, what's the difference?
No they don't. The pension regulator sets very restrictive controls on what they can do with that 125b, because it's neither the company's nor the shareholders' money.
> Google Cloud placed Railway’s production account into a suspended status incorrectly, as part of an automated action. This action extended to many accounts within Google Cloud. As this was a platform-wide action, there was no proactive outreach to individual customers prior to the restriction.
This might be 100% of what google told them.
This is Google we're talking about. This absolutely happened many times in the past and will happen again.
If it were something out of Railways hands, I think they would say something like "We have not yet identified the reason for the suspension, and are awaiting a response from Google".
Generally it takes 30 days past due and complete no contact for anyone before suspension.
Then we got down to a very small number of major suppliers in some areas. Many have moats that deter switching. This gives the big suppliers too much power.
A century ago, this happened with railroads. Which is why railroads were forced to become common carriers. They're required to transport goods for anybody who can pay. Read up on antitrust activity of a century ago. We've been here before.
Oh...my. Just starting out in the industry, are we? Those of us who have been here for a while know reality is very different than newbie hopes and dreams. Once you've been burned for the n+1 time, that optimism will fade.
I'd wait for more details before adjudicating.
In practice, Google has earned the way my priors are ready to believe it's 100% their fault with mighty and sustained effort. Or lack thereof, depending on your point of view.
And in general Google lost any immediate benefit of the doubt status many years ago. Many such stories.
So no, Google doesn't get the benefit of the doubt.
> Google Cloud placed Railway’s production account into a suspended status incorrectly, as part of an automated action. This action extended to many accounts within Google Cloud. As this was a platform-wide action, there was no proactive outreach to individual customers prior to the restriction.
The resulting action should be you have proper disaster recovery, failover, etc.
Not sure I would trust these folks if this is the conclusion they are coming to from this experience. Any cloud provider can/will do this to you.
The best you could hope for is that if there is something fishy with your account, you are contacted by Google to address it.
If you're picking them instead of the underlying cloud provider, but you want all the knows and dials the underlying provider has, you've made the wrong choice.
I know multiple startup founders personally (2 of them are in the current YC batch), and the sheer callousness with which they look at infra, especially from security/scalability/reliability angle is shocking.
I'll personally reserve judgement against GCP (replace with AWS/Azure/OCI/whatever) until we know more.
"However, in this ring, there was still a hard dependency on workload discoverability being tied to the network control plane API that was hosted on the machines running in Google Cloud."
They've gotta be joking me that they deliberately left something so critical under the control of any other entity than themselves. That demonstrates a lack of critical planning and a lack looking at their configuration from a first-principles approach.