I mainly only use them for backups instead of production data
EDIT: As of 2019 there are multiple datacenters, but it doesn't seem like the data is stored redundantly across them
Durability refers to the idea that your data will still be retrievable (ie no corruption) similar to S3's claim of eleven 9s of durability.
Reliability however would be say; the actual availability of the Web UI or API server that you download your data through. If it were down, that wouldn't impact the actual integrity of the data itself.
For anyone interested, any more than about five 9s of reliability is basically impossible anyway when it comes to human intervention. As an example, 6 nines of reliability would allow you 3 seconds of unavailability a year so 250 milliseconds of unavailability a month.
From a users point of view, being "unavailable" includes everything from going through a tunnel and having your mobile connection drop out to a shark biting one of the undersea cables in the middle of the ocean.
As you might imagine, with a human involved, they couldn't even get acknowledge an alert fast enough to meet that deadline let alone actually going about doing any repairs and diagnosis :)
It could be spread over multiple instances and redundant hardware as well but as with any system being touched by humans, it's near guaranteed that something will go wrong eventually.
This helps, too, as it lets us build out services to be more reliable in combination, rather than less reliable. With retries and fail-over, an outage in an entire region may not necessarily result in any user requests failing.
For scale, pre-pandemic our published figures claimed >100M MAU.
I find https://andrewaylett.github.io/multi-burn-rate-calculator/ helpful for visualising error rates -- largely cribbed from the project it's forked from :) but with the tweakables switched around and the time between alert and error budget exhaustion in the tooltip.
It's worth noting that we only evaluate our alerts at most once a minute.
Said Oracle. Then their DNS was misconfigured and their whole cloud went offline for 2 hours [1].
Shit happens, always, at all scales.
[1] https://ocistatus.oraclecloud.com/incidents/qjxllgkywysj
Edit: typo
But perhaps my point wasn't clearly enough made: a claim of "100% uptime" on a service level isn't particularly _useful_ when our users still only see a 99.9% success rate.
Which is to say, individual subscriber lines can have all sorts of faults, but they only affect that subscriber. Trunks between offices can fail, but they only affect a certain number of circuits. The central call processing ability, of the switch to react appropriately to lines changing state and numbers being dialed, for calls to be completed if both stations are available, is what had to meet that target.
I was skeptical when I first heard this, so next time I was in an office that still had a #1 (actually a 1A, it had been upgraded in the late 80s), I talked to the switchman for a bit, and he showed me the downtime counter. It's a mechanical thing like an odometer, and once a second when the switch is executing its main loop, it touches a register that keeps the counter from incrementing. If the processor halts or isn't processing calls for some reason, the counter starts counting.
The switch was installed in the mid-70s, and from the moment it took over for its crossbar predecessor, including an in-place processor upgrade, it had logged less than an hour. Most of that came a few seconds at a time when swapping between active processors during software upgrades, he said. At the time (this was around 2002 I think) it was slated to be replaced with a DMS100, but the replacement activity hadn't commenced yet. I don't know what sort of reliability numbers the DMS machines achieve, but they'd do well to match their predecessors.
If you get better, awesome, but SLA? Too unrealistic.
What's the SLA penalty vs the extra the customer is willing to pay? If I think I can achieve 6 x 9's on a monthly basis, but probably I'll only achieve it 10 months out of 12, I can offer the customer an SLA of 6 x 9's for 100 USD per month.
My penalty can be 50 USD for failing to meet, and then I as a supplier walk away with 10 x 10 + 2 x 50 USD = 1100 USD for offering something I knew I couldn't achieve (consistently).
Not sure what the qualifier “when it comes to human intervention” means, but if I ignore it then five nines is quite standard in certain sectors. For example phone switch SLA (back when I was in the biz) was measured in minutes per decade (as in “cumulative unavailability under 1/2 minutes per decade”). Large baseline power plants can and must run uninterrupted for decades.
Of course it’s a systems issue not a point solution.
I realise in hindsight that I was rambling from the point of view of offering five nines for software that is inherently flaky/unreliable. Companies where developers cycle through and knowledge is lost, technical debt accumulates and systems are used almost counter to their intended purpose (ie Redis as a database)
In that sense, it's like constantly massaging applications to stay alive or at a reasonable level of service and so hence the assumption that someone will be paged and respond in order to preempt a failure or restore service during an outage.
That's why I feel more comfortable storing data across two "unreliable" providers with a lot of physical space in between them rather than one super reliable provider.
You also have to consider that data loss can result of simple things such as an account that gets blocked for stupid reasons. If you want to be safe you always need to have your data with at least 2 providers.