What would a small business do as a contingency plan?
Going multi region on AWS should be safe enough.
If a multi region, multi service meltdown happens on AWS, it will feel like most of the internet has gone down to a lot of users. Being such a catastrophic failure, I bet the service will be restored pretty fast, not in 3 days.
You could go multi cloud though. But when half of the internet struggles to work correctly, I’d not feel too bad about my small business’ downtime.
Additionally, from a "nobody ever got fired for buying IBM" perspective, you're unlikely to catch much blame from your users for going down when everyone else was down too.
edit: I stand corrected. Apparently the S3 outage wasn't global, though its effects were.
Meanwhile, this outage has only really been noticeable to ops teams, since it doesn't affect existing nodes or anything outside GKE. It's definitely concerning and the fix is taking far too long, but as far as global outages go the impact is relatively minor.
Multi cloud is almost always more pain than gain. You’d spend time and effort abstracting away the value that a cloud provider brings in canned services.
Hell, multi region is often more than many workloads need.
Nope.
This was a while back though. Now we depend on a lot more AWS stuff.
Cluster administration and identity management are unique to each provider and fairly challenging to get right.
Then start looking at points of failure and sort them based on severity and probability. Is your own software deployment going to generate more downtime per year than a regional aws outage?
There are formal academic ways to determine what your overall availability is, but don't have those on hand. Suffice to say, it takes significant research, planning, execution, and testing to ensure a target availability. (See Netflix https://medium.com/netflix-techblog/the-netflix-simian-army-... ) if someone says they have 99.9% or better up time, they had better have proof in my mind (or a fat SLA violation payout)
People outsource to cloud providers not because they are cheap, but because managing infra in house is hard. Also move fast and break things.
Read AWS docs about availability, there are availability zones in a region, spread across those to minimize impact. Then test when something goes down. Fix/repeat.
Most companies I’ve been at don’t offer multi region support for their services because it’s too expensive for the service provided even in so-called “price insensitive” enterprises (you can’t just make up a price that’s huge, they do have budgets still) and most of their customers are unwilling / unable to pay more for the extra availability. If your software is designed from the start better, multi region failovers should be fairly inexpensive though. But all the bolted on “multi region” software I’ve seen has been hideously expensive and oftentimes less reliable due to the design being soundly not able to tolerate failures well.
Considering that even tech companies hardly manage to have a pair of DevOps or Sysadmin, running one own infrastructure is completely out of question.
Besides, I’m assuming that the cost savings a small company can get from being billed under a much larger organization account would make up for it. That and having cheap shared netops support.
Of course, that doesn't make them knowledgeable to run stable infrastructure and they will move on as soon as they realize they are being abused to work overnight and week end.
A company that doesn’t want the overhead of an MSP which in my experience is less than the cost of a full time Dev is not a company I’m going to work for. It would tell me a lot about thier mentality.
2 at UTC 0-8
1 at UTC 4-12
2 at UTC 8-16
1 at UTC 12-20
2 at UTC 16-24
1 at UTC 20-04
(Repeat)
The separate account was setup partially on my insistence but it was set up in the same region.
If needed, we could have done VPC peerings across regions. (https://aws.amazon.com/about-aws/whats-new/2017/11/announcin...)
Some services in AWS are also not available in others (I'm quite familiar with AWS Data Pipeline not being available outside the "core" regions like us-east-1, eu-west-1) and having services in one region make usage of resources in another region is a huge change when most developers outside ones with technology literate customers are under the gun to push features out fast over sound design. The matrix of services and configurations necessary to mix and match regions and availability zones is non-trivial if you make extensive usage of AWS services above the IAAS layer.
Also, cross-region VPC peering has a TON of limitations that rather annoying depending upon how well your network has been architected (by default in most companies outside enterprises with a deep bench of network engineers, this would be rated at "complete crap barely better than a typical home wifi network"). Heck, even though I'm non-dumb at networks I have to keep reminding myself of various cross-region VPC limitations when working with refactoring cross-region VPCs like where you can reference security groups, how to propagate Route 53 records, etc.
Any trade comes to me if it's urgent, and I appear more professional as I've got a functioning system.
I might be an chancer running my entire system off an shoestring but being up when everyone else has taken a dive looks good.
https://blog.thousandeyes.com/amazon-route-53-dns-and-bgp-hi...
Ultimately it’s a risk/return decision.
“Is going exclusively with AWS/azure/GCP etc a better decision in reliability, financial and mantainability terms than complicating the design to improve resiliency? And will this more complex solution actually improve reliability?”
If AWS ever screws up, you will be able to continue running the business even if it might take weeks to start over.
For live redundancy, you should have a secondary datacenter on another provider, but realistically it's hard to do and most business never achieve that. Instead, just stick with AWS and if there is a problem the strategy is to sip coffee while waiting for them to resolve it. Much better this way than you having to fix it yourself.
Depends on your definition of small. If it's small enough not to have a dedicated infrastructure team designing multicloud solution, then the contingency plan may be: switch DNS to a static site saying "we're down until AWS fixes the issue, check back later".
Otherwise it depends on your specific scenario, your support contracts, and lots of other things. You need to decide what matters, how much the mitigation costs vs downtime, and go from there.
Without our multi-cloud set up we would have been down for over an hour. In our business this is not an option.
I wish I was only being tongue-in-cheek.
Terraform using AMIs plus chef recipes that work in the cloud and bare metal. Dont use AWS specific services.
This would allow you to spin over to another cloud provider , vsphere or bare metal with minimal work
To answer the original question: It looks like this issue was just a UI bug that affected the console, the service itself wasn't impacted. Events that do impact the service will be contained to a region, meaning you can mitigate it with proper redundancy across regions, no zany multi-cloud solution required.
Disaster recovery by switching to another provider is simple when minimal centos/rhel images are used.
Are you not using any of thier managed services and are you maintaining your own on VMs? If so, you have the worse of both worlds. You’re spending more on hosting and you’re not saving money on letting someone else do the “undifferentiated heavy lifting”.
But I blame most of the cost overruns when using cloud providers on “consultants” who think they are “moving to the cloud” when all they really know is how to setup a little networking infrastructure and know nothing about how to use the developer, Devops, or other hosted solutions.
Most ”consultants” I’ve run across only know how to do a lift and shift and do a one to one mapping of the on prem VMs and networking infrastructure to the cloud. They know nothing about automation, transforming Devops practices, or transforming development practices and architecture.
A lift and shift should only be the first phase.
Thinking that is the reason for consultants is a problem
Note : I worked directly for companies as an employee when building these stacks
The syntax for provisioning these doesn’t work that well for some find and replace to work. Are you using a templater to generate cloud-specific HCL from a template or something? Sounds like a pretty big problem to solve to me and not just something where you can win via discipline.
At the end of the day, changing your underlying infrastructure is so risky and usually not worth the cost benefit analysis, it’s rarely done.
That's a pretty manpower intensive way of operating. I think the fact that you get cloud agnostic this way is probably not worth it.
Even when working in small companies with small infrastructure, I've kept recreation of infrastructure as one of my high priorities (one reason it really bugged me in one job to have to depend on Oracle Databases that I couldn't automate to the same degree.)
In my mind, it's not different from the importance of having, and testing restoration of, backups. If your infrastructure gets compromised somehow, or you find yourself up the creek with your provider, you've got to be able to rebuild everything from scratch.
Then you realize a lot of software and databases can only run from a single instance, zero support for multi regions, and you're not gonna to rewrite everything and resiliency just can't happen.