AWS even have documents telling people how to achieve exactly this! https://docs.aws.amazon.com/wellarchitected/latest/reliabili...
Why don't "premium" service providers like heroku, etc, do this?
I wonder how many services really had 5 nines availability pre-cloud era either. Somehow I feel your view of it being "industry standard" might be slightly rose tinted
The only people who suffer consequences are the staff forced to work overtime performing SEV0 RED ALERT theater. They will work through nights/weekends while the responsible parties tut-tut and "manage" by reading updates they can collate into the post crisis report. After that, everyone participates in the joy of emergency meetings to discuss said report that will be entirely worthless when a completely different part of the system fails the next time. A more reliable HA solution will be worked up by the engineers, finance will estimate implementation costs, and it will be turned down by an executive on the 8th hole green because they don't care about anything except improving profitability so they can hand themselves a bonus.
Not that I'm bitter or anything.
With IaaS that is now easier than ever, yet these so-called cloud service providers dont do any of that - they tie themselves to a single AZ and have ZERO redundancy.
AWS/Azure/whomever "promise" 5 9s uptime. Something goes wrong, you don't get 5 9s, and what do you get?
A system that went down for 4 hours and a $50 rebate on your next bill!
I even gave a link in my comment to what AWS say about this.
Are people downvoting me because they dont read, or what?
And for something like Heroku's managed DBs you can't just achieve 99.99999% availability on a DB without making certain sacrifices. Availability isn't everything past a certain point
What sacrifices you talking about when synchronously replicating to a backup environment? Write latency? How do you deal with that usually? How much is too much? There are strategies to deal with reducing replication related latency depending on level of consistency required.
Im just saying that these cloud service providers offering managed services should be covering all that - they certainly charge as if they do!
And in the case of heroku - and their specific architecture - it is not that complex. Im aware that other cases may vary.
https://devcenter.heroku.com/articles/heroku-postgres-ha
If you don’t pay you won’t get the feature. Given that multi az failover directly impacts their cost that seems pretty fair.
I don't think i have ever worked on a system where 10 mins of data loss is anywhere near acceptable though.
I guess for mostly static pages, or self hosting a blog its ok, although id be pissed if i had to rewrite an article. Makes you wonder who their target market is.
Their "dynos" are ephemeral. They could literally deploy the images to a backup environment hosted elsewhere. Their data services could all be synchronously replicated to that backup environment. And thats it - they dont offer any other core services (and their other services run on the same platform.)
So for (at most) double their infrastructure cost they have another network they can immediately switch over to.
And herokus already soooo expensive. Even if you used a 1-to-1 mapping for ec2 to heroku dynos (which they dont - its multiple dynos per backing instance), you would be looking at 5-10x markup using on-demand instances! Reserved instances are even less expensive. Spot instances can be 5x less again!
I think they could retain their current pricing model and still offer this kind of resiliency - at a minimum.
Fly.io is making strides in this direction, distributing the VMs across multiple availability zones, and routing traffic internally from their multiple geographically distributed POPs - but you need to roll your own DB VMs for multi-az synchronization..
EDIT: seems they do provide managed postgres with synchronized replication now (in beta), neat!
E.g. my electricity provider doesn't.
They ONLY offer fully managed services, which can be backed by the multi-cloud, multi-AZ setup I refer to - but instead a single product outage from a single upstream provider in a single datacenter is affecting all their clients.
This is a regular occurrence for Heroku - and they charge a substantial premium for their "service".
It's S3 that is 5-nines availability.
AWS's published SLA for Compute (which includes EBS) is 4-nines.
One AZ going down is not covered by the 99.99% SLA. AFAIK there isn't any per-AZ SLA, only a single-instance SLA of 99.5%. The effective per-AZ SLA is going to be somewhere between the two.