Google’s “best practices” lead them to deleting an entire customer’s $135 billion pension account [1]. I’m surprised anyone is still reading anything Google writes.
1. https://arstechnica.com/gadgets/2024/05/google-cloud-acciden...
1. https://arstechnica.com/gadgets/2024/05/google-cloud-acciden...
That was a “mistake” that should not have even been possible. If the pension fund had not used a multi cloud strategy the entire business would have been lost. A mistake is not configuring Kafka correctly and losing some data, deleting an entire account should not be given a pass.
“UniSuper, an Australian pension fund that manages $135 billion worth of funds and has 647,000 members, had its entire account wiped out at Google Cloud, including all its backups that were stored on the service. UniSuper thankfully had some backups with a different provider and was able to recover its data, but according to UniSuper's incident log, downtime started May 2, and a full restoration of services didn't happen until May 15.”
Google didn’t recover the data, the customer recovered their data from a different cloud provider.
> This incident did not impact:
> Any other Google Cloud service.
> Any other customer using GCVE or any other Google Cloud service.
> The customer’s other GCVE Private Clouds, Google Account, Orgs, Folders, or Projects.
> The customer’s data backups stored in Google Cloud Storage (GCS) in the same region.
...
> Data backups that were stored in Google Cloud Storage in the same region were not impacted by the deletion, and ... were instrumental in aiding the rapid restoration.
Emphasis mine.
You're quoting, as far as I can tell, an ArsTechnica article that makes unsourced claims about backups being deleted, neither UniSuper's nor Google's previous statements ever mentioned anything about backups being deleted.
I don’t call 13 days a rapid restoration. I also don’t trust Google’s post-mortem documentation more than an independent news organization to be honest about what really happened. Especially while Google is actively gaslighting their users about the errors in its AI search [1].
1. https://www.theverge.com/2024/5/24/24164119/google-ai-overvi...
I'll reiterate, no one involved in the restoration (Unisuper or Google) ever said anything about Google's backups being deleted, in fact basically everything Google and Unisuper have said specific that it was only the VM config that was removed. Ars made up the thing about backups being deleted, which makes an exciting headline, but it doesn't appear at all reliable or based in reporting, just conjecture.
So you just label a reputable news outlet as fake news and then move on..?
“UniSuper had backups in place with an additional service provider. These backups have minimised data loss, and significantly improved the ability of UniSuper and Google Cloud to complete the restoration.”
https://www.unisuper.com.au/about-us/media-centre/2024/a-joi...
Those quotes were pulled directly from UniSuper’s website. Google deleted an account, lost the data, and then took 13 days to recover the pension fund from data stored on another data provider. Maybe you should consider that your employment at Google is damaging your objectivity.
Like I keep saying, nothing in the primary sources supports the claims either that backups or that the accounts were deleted. You've jumped to a particular conclusion, and seem unwilling adjust that conclusion in light of new evidence.
You're making a strong claim, I'm asking you to source it specifically. Instead you're taking a statement from which you can draw multiple conclusions, and picking one (that has been contradicted repeatedly) and telling me I'm unwilling to accept the facts. But they aren't facts, they're your interpretation of vague statements.
I'm happy to accept facts. Facts like "Data backups that were stored in Google Cloud Storage in the same region were not impacted by the deletion" are very easy to understand and difficult to misinterpret. Do you disagree?
Like, even the additional reporting the ars article links to (https://danielcompton.net/google-cloud-unisuper, https://x.com/milesward/status/1792909048830214607?t=Vu__q1h...) basically contradicts both their and your conclusions. You're weirdly hung up on this.
Again, the same quote I already quoted above, direct from UniSuper’s website. They needed to use their backups at a different cloud provider, as GCP’s data wasn’t recoverable. I don’t know why you’re arguing so strongly against this.
Put formally, we have statements that
- 1. A and B exist
- 2. A was used
- 3. A and B were used
The problem is that it was never that good. Anyone who has used K8s at scale will tell you at length how it doesn’t scale. People should stop focusing on tech companies like celebrities and focus instead on domain problems related to their business.
Their internal tooling scales just fine, but all it shares with k8s is some of the underlying concepts. Unlike, say, Bazel, gVisor or Gerrit, which are the real thing (minus some secret sauce tied to internal infra). k8s is good software, and best-in-class when it comes to open source options, but the idea that it is "open source Borg" is silly.
No it isn’t, it’s a solution in search of a problem that is needlessly complex, wastes engineering cycles on what could have been product development, and has violated every principal of orthogonal design.
Maybe if Google focused on doing actual work instead of writing feel good engineering pieces, they wouldn’t have the Google graveyard and an unstable cloud offering that may spontaneously delete multi-billion dollar accounts.
That same sort of thinking is what led to the downfall of yahoo.
Are they?
Are you saying Google’s c suite isn’t out of touch?