Hacker News Reader

Comment by temuze | Hacker News Reader

Friends tell friends to pick us-east-2.

Virginia is for lovers, Ohio is for availability.

Lots of services are only in us-east-1. The sso system isn't working 100% right now so that's where I assume it's hosted.

Yeah I can't log in with our external SAML SSO to our AWS dashboard to manage our us-east-2 resources. . . . Because our auth is apparently routed thru us-east-1 STS

You can pick the region for SSO — or even use multiple. Ours is in ap-southeast-1 and working fine — but then the console that it signs us into is only partially working presumably due to dependencies on us-east-1.

(Correction: you can choose the region but you can't use multiple for the same org.)

Yeah, there are "global" services which are actually secretly us-east-1 services as that is the region they use for internal data storage and orchestration. I can't launch instances with OpsWorks (not a very widely used service, I'd imagine) even if those instances are in stacks outside of us-east-1. I suspect Route53 and CloudFront will also have issues.

Ohio's actual motto funnily kind of fits here:

  With God, all things are possible

Does this imply Virginia is Godless?

Maybe virginia is for lovers, ohio is for God/gods?

Virginia's actual motto is "Sic semper tyrannis". What's more tyrannical than an omnipotent being that will condemn you to eternal torment if you don't worship them and follow their laws.

Virginia and Massachusetts have surprisingly aggressive mottoes (MA is: "By the sword we seek peace, but peace only under liberty", which is really just a fancy way of saying "don't make me stab a tyrant," if you think about it). It probably makes sense, though, given that they came up with them during the revolutionary war.

I mean, most people are okay with dogs and Seattle.

I think I should add state motto to my data center consideration matrix.

This is funny, but true. I've been avoiding us-east-1 simply because thats where everyone else is. Spot instances are also less likely to be expensive in less utilized regions.

I live in Ohio and can confirm. If the Earth were destroyed by an asteroid Ohio would be left floating out there somehow holding onto an atmosphere for about ten years.

Always has been.

Friends tell Friends not to use Rube Goldberg machines as their infrastructure layer.

If you're not multi-cloud in 2021 and are expecting 5-9's, I feel bad for you.

How do you become multi-cloud if your root domain is in Route53? Have Backup domains on the client side?

dns records should be synced to secondary provider, and that provider added to your secondary/tertiary domain dns.

Multi-provider dns is a solved problem.

If you're having SLA problems I feel bad for you son

I got two 9 problems cuz of us-east-1

> ~~I got two 9 problems cuz of us-east-1~~

I left my two nines problems in us-east-1

If you're not multi-region, I feel bad for you.

If your company is shoehorning you into using multiple clouds and learning a dozen products, IAM and CICD dialects simultaneously because "being cloud dependent is bad", I feel bad for you.

Doing one cloud correctly from a current DevSecOps perspective is a multi-year ask. I estimate it takes about 25 people working full time on managing and securing infrastructure per cloud, minimum. This does not include certain matrixed people from legacy network/IAM teams. If you have the people, go for it.

There are so many things that can go wrong with a single provider, regardless of how many availability zones you are leveraging, that you cannot depend on 1 cloud provider for your uptime if you require that level of up.

Example: Payment/Administrative issues, rogue employee with access, deprecated service, inter-region routing issues, root certificate compromises... the list goes on and it is certainly not limited to single AZ.

A very good example, is that regardless of which of the 85 AZs you are in at aws, you are affected by this issue right now.

Multi-cloud with the right tooling is trivial. Investing in learning cloud-proprietary stacks is a waste of your investment. You're a clown if you think you need 25 people internally per cloud is required to "do it right".

All cloud tech is proprietary.

There is no such thing as trivially setting up a secure, fully automated cloud stack, much less anything like a streamlined cloud agnostic toolset.

Deprecated services are not the discussion here. We're talking tactical availability, not strategic tools etc.

Rogue employees with access? You mean at the cloud provider or at your company? Still doesn't make sense. Cloud IAM is very difficult in large organizations, and each cloud does things differently.

I worked at fortune 100 finance on cloud security. Some things were quite dysfunctional, but the struggles and technical challenges are real and complex at a large organization. Perhaps you're working on a 50 employee greenfield startup. I'll hesitate to call you a clown as you did me, because that would be rude and dismissive of your experience (if any) in the field.

I advise many fintechs with engineering orgs from 5 to 5000, 2 in top 100 - none are blindly single-cloud and none have 25 people dedicated to each of their public clouds. The largest is not on any public clouds due to regulation/compliance and have several colocation facilities for their mission critical - they have less than 25 dedicated in the entire netsec org. This is a company that maintians strict PCI-DSS1 on thousands of servers and thousands of services. If you're employing 25 people per cloud for netsec in a multi cloud environment you have some seriously deficient DevOps practices or your org is 5-figure deep and has been ignoring devops best practices while on cloud for a half decade. Hahsicorps entire business revolves around cloud agnostic toolkits. All clouds speak kubernetes at this point and unless you have un-cloudable components in your stack (like root cert key signing systems on a proprietary appliance) you really should never find yourself in such a scenario where you have that many people overseeing infra security on a public cloud. It's been proven time and time again that too many people managing security is inversely secure.

I meant at least 25 people in the DevSecOps role per cloud. Security experts, network/ops/systems experts, and automation (gitlab and container underlay) support.

K8s is one of a hundred technologies to learn and use, and just because each cloud is supported by terraform, you can't swap a GCP terraform writer over to Azure in a day.

And no bank is without their uncloudable components.

This.

Someone start devops as a service please

I imagine there are very few businesses where the extra cost of going multi-cloud is smaller than the cost of being down during AWS outages.

Also, going multi-cloud will introduce more complexity which leads to more errors and more downtime. I'd rather sit this outage out than deal with daily risk of downtime because I'm infrastructure is too smart for its own good.

Depends on the criticality of the service. I mean you're right about adding complexity. But sometimes you can just take your really critical services and make sure it can completely withstand any one cloud provider outage.

Sometimes you can't avoid us-east-1; an example is AWS ECR Public. It's a shame. Meanwhile, DockerHub is up and running even when it's in EC2 itself.

us-east-1 is a cursed region. It's old, full of one-off patches to work and tends to be the first big region released to.

No one was in the room where it happened

This is also a clever play on the Hawthorne Heights song.

One step ahead of you brother: https://twitter.com/rclmenezes/status/1468253515751608325

Didn't us-east-2 have an issue last week?

Can I get that on a license plate?

I wonder why AWS has Ohio and Virginia but no region in the northeast where a significant plurality of customers using east regions probably live.

Reply on news.ycombinator.com