Hacker News Reader

Comment by idlewords | Hacker News Reader

Relevant amusing bit from the Amazon FAQ: "S3 is designed to provide 99.999999999% durability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.000000001% of objects. For example, if you store 10,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000,000 years."

I think my favorite part of that is "on average", as if you will be making repeated ten-million-year trials of this effectively brand new technology.

The point is that once you get into several nines of reliability, really rare events that are impossible to model start to dominate your risk budget.

View on HN

Many people have far more than 10k objects though. Based on that math, if you stored 10 billion objects in S3 (certainly well within the realm of possibility), you'd lose an object on average every 10 years, which is a length of time one can think about.

Your main point (that making high reliability systems more conventionally reliable is building your high wall yet higher still) is definitely valid. But the lossage rate is actually a meaningful number given the extremely large number of objects stored in S3.

I completely agree with you. Where they are intentionally misleading is in pretending that keeping one object for a zillion years is the same as keeping a zillion objects for one year.

Keeping one object for a zillion years sounds like an interesting project, now that you bring it up.

Our data integrity is such that it will survive universal cataclysm and the reduction of all data to a single point of infinite energy.[1]

[1]guarantee applies to a single bit of data[2] [2]bit value must be 1

All I could think of was Hitchhiker's Guide To The Galaxy.

http://www.10000yearclock.net/learnmore.html

Funny enough, its funded by Bezos

Bezos isn't the only one.

The Long Now foundation (http://longnow.org) has some other interesting projects besides the clock.

If you like this kind of thing, I'd suggest reading Neal Stephenson's Anathem. It might be hard to get into it, it might be his most impenetrable book yet but it really pays off later.

Several of Neal Stephenson's books are like this ... I finished the Cryptonomicon earlier this year and have just started to read REAMDE. I often find myself reading the same sentence several times because they are so convoluted ...

Bezos can even make raw matter work extra hours for him.

THERE IS AS YET INSUFFICIENT DATA FOR A MEANINGFUL ANSWER

Ref: http://www.physics.princeton.edu/ph115/LQ.pdf

Every time this story is posted somewhere, I just have to read it end to end.

Relevant: The thermodynamic arrow would reverse during a contracting phase of the Universe or inside black holes.

http://journals.aps.org/prd/abstract/10.1103/PhysRevD.32.248... (S. W. Hawking, 1985 <- Just so you know it isn't a crackpot writing this. It isn't a widely accepted view though)

That's funny, because I had to read it again too.

This is S3 which isn't comparable to Google's persistent disks. S3 is equivalent to Google Cloud Storage which has "99.999999999%" durability as per https://cloud.google.com/storage/

To accurately compare them you'd need to look at AWS EBS: "Amazon EBS volumes are designed for an annual failure rate (AFR) of between 0.1% - 0.2%, where failure refers to a complete or partial loss of the volume, depending on the size and performance of the volume."

The point isn't to compare competing cloud services, but draw attention to the fantastical nature of these durability promises.

I don't think so, PD and GCS are fundamentally different products. PD is like the local disk, and you cannot assume it last forever. while GCS is durable storage and it is safer.

" This outage is wholly Google's responsibility. However, we would like to take this opportunity to highlight an important reminder for our customers: GCE instances and Persistent Disks within a zone exist in a single Google datacenter and are therefore unavoidably vulnerable to datacenter-scale disasters. Customers who need maximum availability should be prepared to switch their operations to another GCE zone. For maximum durability we recommend GCE snapshots and Google Cloud Storage as resilient, geographically replicated repositories for your data. "

In a cloud based infra, one still needs to know the difference. And for the ultra-low numbers, they are from mathematics I think. (# replicas, possibly cross DC/geo)

Glacier is also 99.999999999%.

But those nines are only true if Amazon's software is bug-free. I've lost data on Glacier on my home account. They personally called me to apologize.

So empirically speaking their reliability is far, far worse than stated just based on that one data point.

How so? We'd need to know both how much data cjensen lost and how much data Glacier has in it to tell, neither of which I know from this conversation.

My recollection: around 20GB... however, they clearly were willing to try to get some of it back if that was helpful. It was just a redundant offsite backup for me, so there was no need.

Would have been interesting to see if they had been able to.

Who knows. Maybe his is the last data Glacier will lose in the next [suitably large number] years. That's bad luck, buddy!

I too choose to extrapolate all my losses over the next 100 billion years.

Not now they have fixed the bug.

Yep. They confirmed they understood and fixed it.

And of course there are no other bugs.

At this point, correlation matters. In the unlikely event that I lose at least 1 object, what is the conditional probability of losing another?

(Forgetting about correlation was a big part of the MBS and LTCM financial failures)

And Fukushima.

Assumptions that variables are independent are often very mistaken.

Or the bigger problem: assuming your future reliability looks just like your past. For example, say they have a problem with year 2038. Everything works great until everything doesn't work at all.

They had a bigger problem: modeling with the wrong distribution.

I always think of Feynman's minority report [0] on the Challenger Disaster when these types of risk assessments pop up.

[0] http://science.ksc.nasa.gov/shuttle/missions/51-l/docs/roger...

Indeed "on average" covers the entire period and from initial local-node intake of the data it has to then be propergated to another site to increase integrity probability. So if it takes even 5 seconds to propergate to several sites at different locations then the durability scales from when the clock starts.

Heck once in every 10,000,000 years, the odds of winning the USA lottery are around one in 175,000,000 and yet people partake and somebody wins. So it is always worth looking at odds from another perspective and accept it is either impossible, probably or possible and as we know the return from investment in durability is logarithmic in cost going up and return getting smaller.

Still one of the biggest issues in any datacentre is not just the power, but also the quality of the power and even the slightest noise could and does increase the odds of some electronics going wrong. UPS's (though been many years) will put out a square wave in modulation and the ideal is a perfect sine-wave. It is truly educational to see the quality of many power sources on an oscilloscope. Also equipment on the same circuit can also induce noise, so can vary rack to rack.

Do they spell out what assumptions they're working under? Because, for example, I'm pretty sure that the odds of a civilization-ending asteroid or comet strike in the next year, while quite low, are higher than what's implied by 99.999999999% durability on trillions of objects."

Well, the guarantee needs to be reasonable only as far as you and the courts are alive to sue them. (Corollary: you only need to sustain DDOS attacks when the rest of the internet infrastructure can sustain the traffic).

Why do you specify trillions of objects? They practically promise data loss at that scale.

Also, the typical civilization-ending comet or asteroid is going to destroy 0 or 1 geographically-diverse data centers. Remember that this is a retention promise, not an uptime promise.

Just because it's how many objects they store. But now that you mention it, it has no relevance to the probability of data loss due to global catastrophe, since it cancels out.

If you're proposed reliability is data loss of one in 100 billion per year, that assumes the risk of e.g. global thermonuclear war is no more than that amount, or you're ignoring it for the purposes of the calculation. Which is why I wonder about their assumptions.

As for a comet destroying at most one data center, if it really ends civilization then the data will probably be lost before too much longer even if the facility physically survives.

That's one of the reasons that I'm excited about decentralized storage. Data stored on dozens or hundreds of nodes spread across multiple countries and jurisdictions is much more robust to things like earthquakes, storms, government intervention, and companies going out of business or changing their profitability model. It has a much stronger defence against black swan events.

When you are storing many billions of files consuming many millions of disks, you become vulnerable to black swan events, simply because you are rolling the dice so many times.

I'm quite hyped about ipfs. I think they can deliver in few years.

The irony is that, by that time, Amazon will almost certainly not be around. So doing it in terms of time is actually a different calculation, and this is based on wrong assumptions :)

Well the expectation is different for different players. If Facebook lose 1 photo for 1 customer out of 10M customers, FB wouldn't care. The user may just assume a glitch. For a small business, losing 1 photo for 1 customer out of 10,000 customers can be a big deal, but nonetheless not the worst case. Offer sincere apology, do extra data backup/replication if those data are extremely valuable to customer, and roll out a new service.

Another problem is that any errors in assumptions or omissions made when calculating the odds will be enormously magnified.

I wouldn't trust any of these figures unless they have ongoing efforts to test them empirically. E.g. create distributed databases of 100 trillion objects, mess with them in various ways, and perform correctness checks on them.

You're thinking of average as a frequentist, and Amazon are thinking as Bayesian.

Ever thought about how we can talk about the chance of rain tomorrow? Or the risc that a comet may strike the earth within the next million years?

"just 0.000001% of disk space was permanently affected."

So Google just exhausted their 11 9's for centuries to come.

Which would be embarrassing if this had happened to a product offering such an SLA. Instead, this is just zone-local HDD storage. It will only affect customers that decided to run a system in a single site, making no snapshots or using cross-region replication, etc. Which may be quite a few seeing as there seems to be this misunderstanding that VMs "in the cloud" have physics-defying redundancy attributes. (Not saying you have this idea, just that multiple clients have expressed surprise when their single-VM setup goes down because "isn't this what the cloud is for?".)

Well, single VM is not "the cloud". It's just a VM on some dude's computer. Your clients seem to have been mislead by the incredibly content-free marketing around "the cloud".

Basically s/hosted/cloud-based is what everyone has done.

You are making repeated trials, they just overlap.

Is that durability rate per-... user? project? s3-wide?

Important detail left out.

If there are 10,000,000 projects with 10,000 objects, then a system-wide durability of 99.999999999% would expect to drop 1 object per year.

Reply on news.ycombinator.com