Ask HN: Cloud provider accidentally deletes instances
I work for a small org that utilizes a cloud provider (not top 3, but top 10). Yesterday we noticed that all our instances, attached volumes, sec groups, network resources, from our Test and PROD environments had disappeared. After scrambling we found an audit trail, that showed all the deletes were from an SRE working for the provider. Apparently they were doing "cleanup" and somehow our account number became part of that.
We were told we needed to recreate everything from scratch shortly after opening a ticket. Luckily we host our Terraform and config management on-prem, and a few snapshots and volumes they weren't successful in deleting. Still a lot of manual rebuilding as not everything is automated.
Two of us have been working to recover things for many hours. Lower environments will likely take a lot longer.
Just curious if anyone else has had something like this occur. I have a hard time gauging if this because our provider is lacking, of if this stuff occurs in top three as well. Any recourse with the provider? We are small potatoes (probably 100-120 hosted resources) so don't have a lot of institutional weight to use.
The nonchalance coming from the vendor (at least initially) was pretty infuriating. After that it was putting bodies on a zoom call, which were not much help for the most part.