Hacker News Reader

What’s your experience like at other storage outfits?

I only ask because your post is a bit like singing praises for Cinnabon that they make their own dough.

The things that you mentioned are standard storage company activities.

Checksum-all-the-things is a basic feature of a lot of file systems. If you can already set up your home computer to detect bitrot and alert you, you can bet big storage vendors do it.

Keeping track of hard drive failure rates by vendor is normal. Storage companies publicly publish their own reports. The tiny 6-person IT operation I was in had a spreadsheet. Hell, I toured a friend’s friend’s major data center last year and he managed to find time to talk hard drive vendors. Now you. I get it — y’all make spreadsheets.

There are a lot of smart people working on storage outside AWS and long before AWS existed.

When I worked at Google in storage, we had our own figures of merit that showed that we were the best and Amazon's durability was trash in comparison to us.

As far as I can tell, every cloud provider's object store is too durable to actually measure ("14 9's"), and it's not a problem.

9's are overblown. When cloud providers report that, they're really saying "Assuming random hard drive failure at the rates we've historically measured and how we quickly we detect and fix those failures, what's the mean time to data loss".

But that's burying the lede. By far the greatest risks to a file's durability are: 1. Bugs (which aren't captured by a durability model). This is mitigated by deploying slowly and having good isolation between regions. 2. An act of God that wipes out a facility.

The point of my comment was that it's not just about checksums. That's table stakes. The main driver of data loss for storage organizations with competent software is safety culture and physical infrastructure.

My experience was that S3's safety culture is outstanding. In terms of physical separation and how "solid" the AZs are, AWS is overbuilt compared to the other players.

That was not how we treated the 9's at Google. Those had been tested through natural experiments (disasters).

I was not at Google for the Clichy fire, but it wasn't the first datacenter fire Google experienced. I think your information about Google's data placement may be incorrect, or you may be mapping AWS concepts onto Google internal infrastructure in the wrong way.

I would not lose sleep over storing data on GCS, but have heard from several Google Cloud folks that their concept of zones is a mirage at best.

Yeah, that's definitely true. Google sort of mapped an AWS concept onto its own cluster splits. However, there are enough regional-scale outages at all the major clouds that I don't personally place much stock in the idea of zones to begin with. The only way to get close to true 24/7 five-9's uptime with clouds is to be multi-region (and preferably multi-cloud).

I have experienced many outages that were contained to a specific availability zone in AWS, from power failures to flooding to cable cuts. You are correct that 5 9’s still requires multi-region though.

I think also Google as a whole has pretty good diversity. But Cloud customers demanded regions in big population centers and smaller countries where Google traditionally avoided due to cost reasons. This lead to less redundant sites that were often owned and/or operated by third parties. So in the US and Europe you can probably trust GCP zones quite literally. But other regions (I have heard lots of rumours in the APAC) they may not be quite as diverse as they appear.

> big population centers and smaller countries

Can we stop this dance on HN? Can you just name them, please?

I think most Googlers actually don't know the specifics (I certainly don't know), and if they could, they probably couldn't tell you. It's sort of common knowledge that some of them are like this, but not exactly which ones.

See: that fire in France that took down a whole region

But on the other hand, GCP supports multi-region so that's not nearly as big of a deal as it would be if AWS zones were not sufficiently isolated

Do you mean Google included "acts of God" when computing 9's? That's definitely not right.

11 9's of durability means mean time to data loss of 100 billion years. Nothing on earth is 11 9's durable in the face of natural (or man-made) disasters. The earth is only 4.5 billion years old.

Normally, companies store more than 1 byte of data, and the 9's (not just for data loss, for everything) are ensemble averages.

By the way, I don't doubt that AWS has plenty of 9's by that metric - perhaps more than GCP.

If I were to upload a 50kb object to S3 (standard tier), about how many unique physical copies would exist?

At least 3.

At least 3, in at least 3 seperate datacenters.

According to https://nuclearsecrecy.com/nukemap/ - it'd take at least a 1 megaton warhead to take out two of the ap-southeast-2 datacenters, and over 10MT to take out 3.

I suspect you'd need a lot less than that though, the 1MT warhead would probably take out enough outside-the-datacenter infrastructure to take the entire AZ offline. I don't care too much though, if someone's dropped a warhead that close to home I have other things to worry about than whether all the cat pictures and audit logs survive.

> if someone's dropped a warhead that close to home I have other things to worry about than whether all the cat pictures and audit logs survive.

Speak for yourself. Many of us love our audit logs and show them to strangers whatever we can.

I’m picturing you having a slideshow of audit logs that you make guests to your home sit down and watch with you, like the vacation pictures slideshow of old.

Yes but my audit logs are special and everyone just loves them, although they playfully act bored

scrolling through them like it's the matrix can't be all that boring!

Awwwww! Check out this cute little IAM audit log! Look at its funny little fizzy privilege escalation! I just want to scratch it's belly until it p0wns the whole prod deployment.

Oh, like you don’t?

We’re among friends here.

I think lees. Why AWS need to store 3 times if they can use Reed-Solomon algorithms (or similar) and decrease this number to 2 or 1.5 and save a lot of storage space

Reed-Solomon would allow you to lose any one of the three and recover. Losing two would be catastrophic.

AWS's guarantee is that you can lose two of the three copies and still have all the data. You can't do that without three complete copies.

1MT will take out the infrastructure to make the data not Available. However the data is still in the 3rd datacenter, making it still accessible therefore no compromising on Durable but yes we don't need the cat pictures when that is close to home :)

I'm guessing that even though the 3rd datacenter is about 35km away from the other 2, and so the building isn't in the expected destruction zone of a 1MT warhead, the damage to the city's electricity/water/network infrastructure would take the 3rd datacenter offline as well - so while your cat pictures are probably still in existence on the no-longer-spinning-rust there, they'd be unaccessible for quite some time.

Probably for just a short period of time before it’s erasure coded

0. user error (deletion or overwriting a file they regret later, possibly much later)

-1. government, the historical cause of most data loss

1½. google canceling the product and deleting all the data, as they did with google+

9's are useful when they're backed by an actual SLA - like GCP Cloud Storage and AWS S3 availability SLAs. Neither one commits to any durability SLAs whatsoever so I wouldn't put any stock into the 'eleventy nine nines' durability claims.

> As far as I can tell, every cloud provider's object store is too durable to actually measure ("14 9's"), and it's not a problem.

How can you tell that if it's not measurable?

As far as I can tell the '11/14 9s' durability numbers are more or less completely made up. That's why AWS doesn't offer any actual durability SLA for S3, only a 99.9% availability SLA[0].

[0] https://aws.amazon.com/s3/sla/

buddy you are mixing availability and durability

Sorry I am not buying the personal anecdote as the public numbers from both orgs tell a different story. When reliability and long term support come up in conversation, Google is not a name to reach for.

Note that I said "durability," not any other reliability metric. GCP is pretty well-known for its outages and abysmal support. It's a reputation they want to change, but they did earn it.

However, Google is very good at not losing data.

Yup still not buying it. People who want their production stable, secure, and durable do not choose Google.

it's well known and not debatable that Cinnabon is fire

"Checksum-all-the-things is a basic feature of a lot of file systems"

"A lot"? Does anything but ZFS and maybe btrfs do this? Ext4 anf XFS — two very common filesystems — still don't have data checksums.

Bcachefs, and LVM also has a way to do it.

Unfortunately I’m not aware of any filesystem that does it while maintaining the full bandwidth of a modern NVMe. Not even with the extra reads factored in; on ZFS I get 800 MB/s max.

ZFS should absolutely be able to go faster even with lz4 compression I get writes above 5 GB/s on a older 32-core EPYC CPU. And that is with mostly random and already compressed data. And that write speed is a limitation of the RAIDz2 on top of not the fastest drives (6 PCIe 3.0 intel U.2 ones from 2 years ago).

Maybe I've configured something wrong, then. I'll do some tests.

This was a few years ago, but blob storage on GCP had a global outage due to an outage in a single zone. That, among numerous other issues with GCP, lost my confidence entirely. Maybe it’s better now.

As a point of order: Not all Cinnabon locations make their own dough. The one I worked at the summer of 2001 made their own dough and frosting that summer, but switched to premade rolls partially that holiday season, and fully to premade rolls and frosting by the 2002 holiday season.

Also: you had to be eighteen or older to operate the mixer. It was something like a 60 quart machine, and all the recipes were pre-programmed for time and power with pauses to change from the paddle to the whip if needed.

The beauty of proprietary systems is that the only information we can ever expect to get about how they’re built is the biased information we get from the builders of those systems.

"AWS' availability zone isolation is better than the other cloud providers."

Not better than all of them.

A geo-redundant rsync.net account exists in two different states (or countries) - for instance, primary in Fremont[1] and secondary in Denver.

"S3 even operates at a scale where we could detect "bitrot""

That is not a function of scale. My personal server running ZFS detects bitrot just fine - and the scale involved is tiny.

[1] he.net headquarters

Agree.

> S3 even operates at a scale where we could detect "bitrot" - random bit flips caused by gamma rays hitting a hard drive platter (roughly one per second across trillions of objects iirc).

I would expect any cloud provider to be able to detect bitrot these days.

I think the point the OP was trying to make is that they regularly detected bitrot due to their scale, not that they were merely capable of doing so.

Ah, thank you. This makes more sense. And I think I remember reading about it once. Apologies for the misinterpretation!

Everyone with significant scale and decent software regularly detects bitrot.

Backing up across two different regions is possible for any provider with two "regions" but requires either doubling your storage footprint or accepting a latency hit because you have to make a roundtrip from Fremont to Denver.

The neat thing about AWS' AZ architecture is that it's a sweet spot in the middle. They're far enough apart for good isolation, which provides durability and availability, but close enough that the network round trip time is negligible compared to the disk seek.

Re: bit rot, I mean the frequency of events. If you've got a few disks, you may see one flip every couple years. They happen frequently enough in S3 that you can have expectations about the arrival rate and alarm when that deviates from expectations.

> The neat thing about AWS' AZ architecture is that it's a sweet spot in the middle

What may be less of a sweet spot is AWS' pricing.

Sending the data to /dev/null is the cheapest option if that’s all you care about.

Seems the snark detector just went off :)

Back on topic, I'd hope all of us would expect value for money for any and all services we recommend or purchase. Search for "site:news.ycombinator.com Away From AWS" to find dozens of discussions on how to save money by leaving AWS.

EDIT: just one article of the many I've read recently:

"What I’ve always found surprising about egress is just how expensive it is. On AWS, downloading a file from S3 to your computer once costs 4 times more than storing it for an entire month"

https://robaboukhalil.medium.com/youre-paying-too-much-for-e...

And that is egress which works as expected, unlike the AWS S3 denial of wallet attack...: https://news.ycombinator.com/item?id=39625029

> the network round trip time is negligible compared to the disk seek

Only for spinning rust, right?

Yes, which is what all the hyperscalers use for object storage. HDD seek time is ~10ms. Inter-az network latency is a few hundred micros.

> They're far enough apart for good isolation, which provides durability and availability

It can't possibly be enough for critical data though, right? I'm guessing a fire in 1 is unlikely to spread to another, but could it affect the availability of another? What about a deliberate attack on the DCs or the utilities supplying the DCs?

Yes, if a terrorist blows up all of the several Amazon DCs holding your data, your data will be lost. This is true no matter how many DCs are holding your data, who owns them, or where they are. You can improve your chances, of course.

There have been region-wide availability outages before. They're pretty rare and make worldwide news media due to how much of the internet they take out. I don't think there's been S3 data loss since they got serious about preventing S3 data loss.

> but could it affect the availability of another

Availability is a different beast than durability. I think people are paranoid here about durability instead of availability.

S3 advertises four nines availability and 12 nines durability.

FWIW, both AWS S3 and GCP GCS also allow you to store data in multi-region.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/MultiR...

https://cloud.google.com/storage/docs/locations#consideratio...

Yes, but S3 has single region redundancy that is better than GCP. Your data in two AZs in one region is in two physically separate buildings. So multi-region is less important to durability.

How does the latest ZFS bug impact your bitrot statement?

I mean, technically it’s not bitrot if zeros were accidentally written out instead of data.

Probably none because they didn't update to the exact version that had the bug

Checksumming the data is not based out of paranoia but simply as a result of having to detect which blocks are unusable in order to run the Reed-Solomon algorithm.

I'd also assume that a sufficient number of these corruption events are used as a signal to "heal" the system by migrating the individual data blocks onto different machines.

Overall, I'd say the things that you mentioned are pretty typical of a storage system, and are not at all specific to S3 :)

The S3 checksum feature applies to the objects, so that’s entirely orthogonal to erasure codes. Unless you know something I don’t and SHA256 has commutative properties. You’d still need to compute the object hash independent of any blocks.

Source: https://docs.aws.amazon.com/AmazonS3/latest/userguide/checki...

It's not entirely orthogonal; RAID5 plus stripe-level CRC (or better) can reliably correct bitrot at any single position in a stripe whereas RAID5 alone can only report an error. My guess is that S3 and other large object stores have the equivalent of stripe-level checksums for this purpose.

I’m positive something like this is the case. Yet it’s entirely orthogonal to the object hash in the user facing feature, which would need to be computed separately.

For append-only or write-once objects or for BLAKE-3 and other fully parallelizable hashes it's possible to store the intermediate hash function state with each chunk or stripe so that the final bytes of the data, once the hash is finished, yield the user-facing checksum as well.

> customers would beat us up over pricing compared to GCP blob storage, but the comparison was unfair because Google would store your data in the same building

I don’t think this is true. Per the Google Cloud Storage docs, data is replicated across multiple zones, and each zone maps to a different cluster. https://cloud.google.com/compute/docs/regions-zones/zone-vir...

Google puts multiple clusters in a single building.

Seems you’re right. They say each zone is a separate failure domain but you kind of have to trust their word on that.

Flashback to that Clichy datacenter fire near Paris...

Zones are about correlated power and networking failures. Regions are about disasters. If you want multiple regions, Google can of course do that too:

https://cloud.google.com/storage/docs/locations#consideratio...

> Believe the hype.

I'd rather believe the test results.

Is there a neutral third-party that has validated S3's durability/integrity/consistency? Something as rigorous as Jepsen?

It'd be really neat if someone compared all the S3 compatible cloud storage systems in a really rigorous way. I'm sure we'd discover that there are huge scary problems. Or maybe someone already has?

But they asked if the claims were audited by a unbiased third party. Are there such audits?

Alternatively, AWS does publicly provide legally binding availability guarantees, but I have never seen any prominently displayed legally binding durability guarantees. Are these published somewhere less prominently?

> Alternatively, AWS does publicly provide legally binding availability guarantees, but I have never seen any prominently displayed legally binding durability guarantees. Are these published somewhere less prominently?

It's listed prominently in the public docs: https://aws.amazon.com/s3/storage-classes/

I read that page and it does not provide any contractual durability guarantees as far as I can see. It provides "designed for availability" and then contractual availability SLA guarantees. It provides "designed for durability", but presents no contractual durability guarantee as far as I can see.

Given that their lawyers clearly indicate that "designed for availability" is not what they are contractually obligated to provide, only the letter of the SLA does that; "designed for durability" is similarly a marketing statement that does not incur any contractual obligations. Is there some specific statement in that document that I am missing which indicates that data durability is not fully at their convenience?

SLAs are more of a financial construct than anything else. Once the payback cost of missing an SLA is built into the contract then it just becomes a conversation about money. I've been at plenty of shops that obviously tried to hit the SLA but if it was missed it just became a financial issue which helped smooth over what otherwise might have been a trust buster.

I would never ever think of an SLA as anything more than a financial commitment - if you think more of it you'll eventually be in a world of hurt.

Obviously, as I made no mention of specific performance in the event of breach of contract the remedy for failure to meet contractual obligations would be damages.

The question is: What level of durability does AWS contractually guarantee where a failure to provide that level results in a breach of contract that may incur damages and where, specifically, in the documentation do they specify that?

My first job was at a startup in 2012 where I was expected to build things at a scale way over what I really had the experience to do. Anyways the best choice I ever made was using RDS and S3 (and django).

Not a public cloud, but storage at Facebook is similar in terms of physical infrastructure, safety culture, and scale.

I also worked at AWS, but not in the S3 team. However, I was Tech Evangelist and met with literally thousands of customers over my 6 years tenure. S3 was one of the hottest topics, but I got a sense of how good and robust it was directly from these customers.

What you say resonates really well with me, and what I've heard during these years.

> and bigger events like natural disasters

Outdated anecdata: I've worked for a company that lost some parts of buckets after the lightning strike incident in 2011, which bumped the paranoia quite a bit. AFAIK same thing couldn't happen for more than a decade.

Google discovered random bit flips caused by gamma rays.

Correct me if I'm wrong but bitrot only affects spinning rust since NAND uses ECC?

If you see this I wonder if S3 is planning on adding hardlinks?

Pretty much any modern storage medium depends on a healthy amount of error correcting code.

Nand is constantly moving around your data to prevent it from bit rotting. If you leave data too long without moving it, you may not be able to read the data from the nand.