Hacker News Reader

Comment by Art9681 | Hacker News Reader

No disaster recovery plan? No offsite backups? Someone failed to applied the most basic principles that have existed for decades.

View on HN

That depends on the data. If this is EBS or single-AZ S3, then from Amazon's perspective this was correct. Backup responsibly (for any data that does need to be backed up) lives with the customer, and Amazon has no way of knowing about that. EBS data data is unrecoverable, and that's what's reported.

Now if this was multi-AZ S3 or whatever then this would be significant.

The article does not tell us what products were impacted.

I was unaware that Amazon even sold single AZ S3. 20% discount. Doesn't seem worth it. By the time I commit to purchasing S3 space, it has to be important data.

I get that S3 is convenient and reasonably performant, but it is not cheap at all.

That’s simply not true. I use S3 (well GCS mostly) for data that I wouldn’t be upset if it’s lost. And I pay the zonal discount for it.

Google internally has lots of possible redundancy levels for data.

They don't sell any of the lower and less reliable levels to the public, I suspect simply because the reputational damage from losing user data is so bad, and the news will take no notice of the fact the user got a discount for less reliable storage.

Most of Google's customers wouldn't know how to choose anyways, if these were exposed. My memory was quite hazy but I recall having a discussion with my colleague on choosing which Reed–Solomon code for our Colossus files, and apparently the choice was down to RS(8,3) or RS(9,3). I don't think even as Googlers we really had enough information to make an informed choice. Comparatively it was much easier to decide which cells to use for multi-location replication in Placer.

You call it "discount" but it's a 20% discount on a 10x inflated price, so it's an 8x inflated price

Single AZ S3 has other benefits. The point isn't the price, it's that it's _highly performant_ since you can keep all of your reads in the same AZ

Yeah agreed - any ephemeral stuff I need is generally in DynamoDB - S3 (and database) are for permanent storage.

It's a great service for large caches. For example, we process a lot of imagery that we download from third-party providers. We save a lot of latency by storing the data in a single-AZ S3.

If it dies, we will just have to re-download the data.

> EBS data data (sic) is unrecoverable, and that's what's reported.

I don't see where this is reported? TFA does not mention EBS. In fact, TFA seems to be nigh content-free, beyond "AWS (allegedly, and is uncited) says they cannot restore some data."

> The article does not tell us what products were impacted.

… right … which conflicts with EBS being what's reported …

(I would agree with your point that if EBS, or some AZ-level data was lost, then, yeah, that's the contract.)

The more dramatic contingency you have to plan for, the more expensive the plan gets.

Earlier this week I mentioned that if we lose enough data centres to bring our operation down, the first items in the to-do list becomes securing weapons, vehicles and fuel.

"Daddy, where were you when the flames reached our house?"

"I was in the office, reviewing Terraform plans"

I had the same discussion with a manager about the backups of financial contracts for cleaning school facilities.

He just couldn't get past the notion that if the six copies in four buildings across two states were all simultaneously physically destroyed, then most likely there are also no more schools left standing, and hence the contracts to clean them are null and void. Also, payment is now in booze and ammunition, not dollars.

> to-do list becomes securing weapons, vehicles and fuel.

I toured a datacenter once back in the early 2000s and they showed me 30 days of generator fuel storage. When i asked them why 30 days and not 35 they replied "we're such a major customer of both electricity and fuel that if we don't get electricity or fuel for 30 days there's way bigger problems than your website not being online" hah.

That's probably already true for 7 days or less

7 days of unreliable electricity wouldn't be unheard of for a very large storm

Large storms regularly result in some customers with lack of utility power for more than 7 days for some customers. When storms take out major transmission lines and roads and bridges, you can end up with some pretty lengthy outages, and fuel deliveries will also be difficult.

Look at data center responses from Hurricanes Katrina and Sandy. This guy [1] was onsite for Katrina. Lost utility power on August 29. They did have some access to fuel on Sep 1, but pretty spotty until maybe the 3rd. Looks like power started coming back in some places Sep 8, and maybe widely restored on Sep 14th.

I would say, by 7 days in you'll probably have a good idea of if 30 days might not be enough.

[1] https://web.archive.org/web/20061206181753/http://interdicto...

I need to bring this out:

https://craphound.com/overclocked/Cory_Doctorow_-_Overclocke...

You also need to understand whether the generator backup actually runs everything. Where I work it doesn't. Only "essential" systems get backup power.

And if the data center is more than about 5 years old it almost certainly was not planned with adequate backup power to run racks of GPUs.

I had a similar discussion in the low 1990s at a large electronic manufacturer that had a pair of Unisys mainframes, one in São Paulo and another in Manaus, in the Amazon region communicating over a satellite link (two large 3m dishes on both places). There was one question of what happens if both fail at the same time, and I pointed that anything that takes out São Paulo and Manaus at the same time is a civilisation ending event, and we shouldn’t worry too much about that.

Anyway, we had a load shedding agreement with a large bank across the street from São Paulo and we could switch over to their Unisys mainframe in a matter of minutes, and vice versa.

Offsite to.. where? Sea? Data residency in Gulf states is very strict and basically nothing is leaving the countries

Typically 300 miles geographically but could be hard in some Gulf States

In a Gulf State 300 miles is still within ballistic missile range and any belligerent is going to target both places if at all.

Strictly speaking from a missile defense perspective there's an argument 2 sites are a waste of valuable interceptors.

They're likely going to target two datacenters, not the datacenters + your medium sized company's office NAS and the safe in the office manager's home.

(Encryption handles confidentiality concerns.)

> (Encryption handles confidentiality concerns.)

Which is why data residency is such a stupid concept.

Yes and no. For example if you are doing Azure, technically Azure can see tenant traffic I believe and you need to use both a Platform Key and CMK for data rest. VMs need encryption at host turned on too.

There is nothing to say that a determined adversary may still get at your data so it needs to stay in country.

Yeah same with AWS - they say they can't see my custom KMS key but... this stuff all lives on their servers, not on my servers. AWS definitely have the ability to see my KMS keys and decrypt my data but I assume that they won't unless a judge tells them to.

A datacenter not owned/run by a major US or Israeli company seems like it might be a good first step.

Do you think they could ask for a backup

But that is something the customer needs to consider. AWS doesnt offer that as standard if your data is in one zone, and during a war even multiple zones in the same region may not be sufficient.

If you had data at two facilities in different countries hundreds of miles apart (about 250 miles between Dubai and Bahrain), that would count as offsite backup most of the time.

Certainly, this event will inform people's disaster recovery plans, but when you're also looking at data residency requirements, small countries, and state level military action against your hosting provider, it can be hard to keep your data.

Nothing is ever real-time. Eventual consistency leads to some data are not backed up.

You talking as if this is some mom-and-pop shop that you run.

That isn't recovery from AWS's point of view. If the customer has data in another region, thats great for them but AWS isn't really a part of that, AWS doesn't know which data is fungible in every case. Sure they have some data is replicated, what they can't recover is the data THEY do not replicate.

If a AWS customer chooses to store their data in a single AZ, that is a design choice. AWS is not taking a daily copy of a entire regions S3 cluster and driving it to some warehouse for a "just in case" situation. That is why Multi-AZ exists.

Isn't S3 claiming eleven nines of data durability?

https://aws.amazon.com/s3/storage-classes/

"Additionally, S3 stores data redundantly across a minimum of 3 Availability Zones by default, providing built-in resilience against widespread disaster."

I wonder if "can't restore some data" includes any S3 data?

I'd expect to lose EC2 instance EBS data in the event of a datacenter being destroyed, but I kinda assume I wouldn't lose S3 data? Now I'm wondering if RDS backups are more like EBS or S3...

1/f noise strikes again

Did local laws permit?

Not all data is allowed to leave all countries.

what a silly notion.

You apparently don't do business out here with the unwashed masses where "whadda mean with all that nonsense? It's cloud...it's by definition safe[1][2]!" is an all too common preconception.

[1] That's a quote, including the Boston accent. [2] The only one I had that was better was a C-level who said "why are you asking for all this money for security in Azure. It's Microsoft so it's already secure.". That, too, is a quote.

Reply on news.ycombinator.com