Hacker News Reader

Comment by xwowsersx | Hacker News Reader

Oof, that sucks and I feel for you. That said...

> setting up in a new region would be complicated for us.

Sounds to me like you've got a few weeks to get this working. Deprioritize all other work, get everyone working on this little DevOps/Infra project. You should've been multi-region from the outset, if not multi-cloud.

When using the public cloud, we do tend to take it all for granted and don't even think about the fact that physical hardware is required for our clusters and that, yes, they can run out.

Anyways, however hard getting another region set up may be, it seems you've no choice but to prioritize that work now. May also want to look into other cloud providers as well, depending on how practical or how overkill going multi-cloud may or may not be for your needs.

I wish you luck.

View on HN

"Multi-cloud from the outset" is probably the single-worst generic cloud advice that I think anyone could be given. In professional cloud consulting the rule of thumb is to do one cloud with excellence until you even think about another one. And even that is really just kicking the conversational can, as both becoming excellent and actually needing multi-cloud combined is one-in-a-billion.

That pretty much binds your hands since in our experience the one provider who can do “one cloud with excellence” is AWS.

(As an aside I also agree that multi cloud from the get go is a YAGNI violation. Just keep in the back of your mind “could we have an alternative to this?” when using your provider’s proprietary features.)

That generalizes to every kind of lock-in: have a viable escape plan, but only execute it if you need or it becomes cheap enough that it won't harm you.

Just having the plan is already expensive enough.

My experience is the opposite: AWS has more features on paper but most of them exist only to tick a checkbox. Azure has more integrations between their offerings, as well as Azure Active Directory, and Microsoft 365.

And Active Directory integrates horribly with everything outside Microsoft.

Azure Active Directory has both SAML and OpenID Connect endpoints… what’s missing?

We use it across many non-MS services without issues. Care to expand?

We’ve had reliability and availability problems esp with azure and also Google. less with AWS.

None are ideal.

Why do I want AD or Microsoft 365?

You personally? No idea. You probably don't. But many (most?) businesses use AD and Office and aren't particularly interested in migrating to alternative solutions.

Yes, Microsoft focuses on the customer (corporate IT), not the user.

This is how the iphone was able to nuke windows phones which were designed to meet the needs of IT

I mainly work in startups. None of them in recent memory have bothered with AD or Office. Okta and Google Workspaces take their place.

Those MS products have an “IBM mainframe” problem. New businesses won’t choose them.

That’s why I say “why do I want them?” If I was starting a new business I’d have no reason to use them.

O365 = teams, docs, outlook, etc. workspace tools

AD = identity, access, privileging, SSO

I know what they are. No company I’ve worked for recently used them.

In my mind they’re legacy business products.

Sure, most businesses use them, but I don’t necessarily believe that is a forever thing. At one point most businesses had mainframes.

Refuse reality at your own peril. These tools are actively used at scale across most enterprises, and likely will be for the majority of your career. More than 95 percent of Fortune 500 companies use Azure/O365.

How many Fortune 500 companies were founded in the SaaS era?

I'm going to guess that the 5% that don't use Azure/Office 365 represent newer entrants to the Fortune 500.

71% of Fortune 500 companies use mainframes [3], and yet they are considered a dead technology with essentially no future. Do you know anyone or anyone who knows anyone who learned how to develop on mainframes in college in the current millennium? It sure wasn't part of my CS curriculum!

Small businesses represent almost half of US economic activity [1] and represent 99.7% of firms with paid employees [2]

[1] https://advocacy.sba.gov/2019/01/30/small-businesses-generat...

[2] https://cdn.advocacy.sba.gov/wp-content/uploads/2021/12/0609...

[3] https://www.precisely.com/blog/mainframe/9-mainframe-statist...

Just to steel man your statement: You should strive for excellence in deployment with all providers (dabble) but have your initial core setup on one cloud (YAGNI principle) until when multicloud capacity is needed (at scale)

Completely agree, though certain aspects, such as running on k8s or Docker might make it easier to switch if you ever decide to, versus say, being tightly coupled with many bespoke cloud products.

My philosophy is to make switching to a new cloud possible. It doesn't have to be easy. We just shouldn't nail our feet to the floor.

Or you could just deploy on metal, which will be cheaper and sufficient for vast majority of cases. Plus you can always migrate to VMs with relatively low hassle.

If you use generic enough services (container hosts, load balancers, VMs, object stores, even hosted SQL DBs, etc), then the multi-cloud journey is not that hard. The challenge comes when you have build a whole architecture on top of some AWS magic that simply does not have an easy alternative in the non-cloud world.

Surely it would depend on the reliability demands of your product.

On top of which, startups often don't have that luxury; you have often need to ruthlessly prioritise your effort.

True. When Nextel was bootstrapping in the 90's, a VP said, "We have to buy gas for the car now. Later we'll buy the seat belts"

Who are you quoting? I said "multi-region from the outset" and later acknowledged that multi-cloud would probably be overkill.

This was exactly what I was thinking, its amazing what people read into, probably myself included.

You always only need another one when everything has gone to shit, either from failure or cost from vendor-lock-in, so drinking your chosen providers Kool aid equals taking the reactive route and scrambling to rearchitect when the issues hit.

Multi-cloud is really not a big deal. Main nuisance is billing differences, followed by slight variations in e.g. Terraform config.

Totally agree. If the service you're providing is so important, build your system so it can fly on one engine or at least land safely. Multi-cloud is the equivalence of trying to transfer all of your passengers to a different aircraft mid-air.

Multi-cloud should only be for mission critical infrastructure. Very little infrastructure is mission critical. Most other use cases can be temporarily wallpapered over with an "Under maintenance" page unless there's a good reason otherwise.

Multi-cloud introduces more risk than it prevents. Which is why things like simulated failovers and BCP testing is constantly required.

I'll reply to my own comment in response to a since-deleted reply that went something to the effect of "this is terrible advice for a young startup trying to get to product market for":

I'm totally on board with the idea of being scrappy and taking shortcuts in order to get to PMF as soon as possible. However, it seems the proof is in the pudding here. If you can't service customers due to lack of compute resources, you can't get to PMF.

Also, yes there are certain infrastructure and network topologies that would absolutely be overkill for a young startup. I don't think multi-region is one of those things. I don't have experience with Azure directly, but on every other cloud providers, going multi-region is not something that requires huge amounts of time or resources. You just need to be mindful of it from the outset. And if you decide not to be, then at least be intentional and conscious about the risk and have a plan in place for what happens when you get bit by deciding not to go multi-region.

GDPR might be a problem here. But this brings us to an important point: this is not your infrastructure, but someone elses.

Good point, I didn't think about that.

There are quite a few European regions, but we don't know how the others stand with their computing limits...

GDPR isn't really related to the infrastructure, and isn't a problem if you built your product knowing you'll need to conform. Shopify is GDPR compliant, for all merchants, and runs on Google Cloud in multiple regions.

There are specific requirements in Germany that require user data to not leave the country. I believe that was what OP was referring to.

Genuine question: is knowledge on how to do this well known? Without that accessibility, I'm picturing folks operating in EU being unwilling to take the risk of not being compliant and just hosting everything in a single region.

They’re wrong.

GDPR is not some boogieman, it can be pain to do on existing products that were build pre GDPR, but if you are starting new project, being GDPR compliant is pretty straight forward and not hard/time consuming, unless you are explicitly trying to do something shady*

*privacy invasive that GDPR is explicitly set up to make harder, so duh

Privacy shield very much matters where your servers are. EU cracking down hard on extra territorial transfers in the past year with more to come.

Also, lots of companies assert GDPR compliance via magical thinking. They most often are wholly wrong. Shopify can say whatever they want, but there’s no certification body.

Source: I’m the person who evaluates and builds compliance systems for a range of services you almost definitely use.

I'd add to this by asking: how much more PMF can you get when you have a two week horizon of new customers before you literally run out of compute resource in a major cloud provider data centre?

Sounds like customers are coming in thick and fast.

If this is the dynamic and the company can't spare a few weeks to solve it, something has gone seriously wrong in a very interesting way.

Also, n8n arguably has product-market fit so the advice was impertinent to start with…

I take offense with you comment. It's not the first time I'm hearing about multi-region/multi-cloud in online tech forums, however reality doesn't match.

I don't want to be snarky, but when large service providers like AWS have their own crossregion downtime because one snowflake of a service in us-east-1 is down, I kind of dismiss the virtue signaling of high resilient multi-(az/region/cloud) ever existing in practice.

If you can somehow have a separate database per region/cloud, sure, I can understand that, but if you have to shard your database across many clouds, I'd dread having to tame such a beast, especially within a startup.

> dismiss the virtue signaling of high resilient ...

So you're saying it's impossible to improve reliability from 97% to 99% because you can never make it to 100%.

If your single-AZ, single-region cloud is not giving you 3 or 4 9's of reliability out of the box, you are using the wrong cloud.

Multi-AZ and multi-region add complexity and cost much more quickly than they add reliability.

Sometimes it is worth it. Sometimes it is not.

Depends on your needs, but having your data & database multi-az to ensure durability can avoid you having to restart from backups. I'm thinking about an old AWS incident were they actually lost EBS data: https://www.bleepingcomputer.com/news/technology/amazon-aws-... Also make sure your backups are in a different AZ (thinking about OVH ...) or region or even at a different provider.

Thanks a lot! You are totally right, it is for sure something we will find a solution for. But honestly, do I not want to. As a startup, you have very few resources and deliberately place some exact bets. Deprioritizing everything to work on something for a long time that was not prioritized, just to then end up again where you were before (a working cloud solution) is the last thing any startup should be forced to do. Anyway, it seems like we do not have much choice here.

I hear you. It's not a fun position to be in. And sometimes you're correct to take calculated risks and maybe the expected value was positive here, despite what ended up happening.

Without knowing the details about your services and infrastructure, it's hard for me to know what's involved in going multi-region now. Are you sure it's such a a gargantuan effort? I would've thought one person working full-time on this for a week or two would be enough, but again I don't know the details of your setup.

One option would be to pay a consultant who is an expert in Azure/cloud stuff to come in and help. May not be cheap, but could be a lot better and quicker for you and better for the business, especially if none of you are really big experts in Azure.

I've been here before (I think)...had to wear many hats and scramble to make sales, build the tech, act as de facto DevOps person even without a lot of experience doing it, etc. That is the way, but stuff happens.

Happy to chat about specifics if you want to bounce ideas off of me or go through your particular situation. Can't promise I'll have concrete advice, but happy to talk it through.

Thanks a lot, is really super nice of you and appreciated! Luckily we have somebody very knowledgeable on our team. Will tell him to reach out if he wants to have a peer to brain-storm some ideas.

Glad to hear you have the right people. Good luck, my friend.

I disagree with the other poster you should have been multiregion from the start. It adds a load of complexity and failure cases for early stage Startups.

Very poor position to be in, apparently this happened in azure UK recently too.

I do not think is a bad idea to be multi-region from start. For the most part Azure has at least two regions in each country (Germany North/West Central, UK South/West, Sweden South/Central, Norway East / West, UAE North / Central, France Central / South etc....)So if stuff happens being able to bring up your service in a different region in the same country could be helpful. I do not know specifics but it seems to me that having an abstraction layer on top of the region is not that hard to do (most of Azure services are supported in all regions). OF course, is a lot easier if done at the outset. Being forced to do it quickly and with little notice is no fun at all....

I feel for you. Also it sucks to be in this position.

Let the scar you get from this is be a learning experience, hopefully you will not fall into the same trap again to trust this company.

In my career I'm in a place where anyone suggesting I do work on Azure gets an instant doubling of my asking day-rate and I really hope the will be put off and find another victim for this gig.

That said, another learning experience would be to use terraform or something (tbh for azure the only sane thing is terraform, ARM templates are just garbage). Having terraformed your one region switching to the other would be much easier, tho not trivial.

Is it that much cheaper for you to build a new region on Azure versus getting setup on AWS?

If you rely on Kubernetes for orchestration and have minimal cloud API dependency, it may be worth that evaluating this option.

Also, do you have a TAM associated with your account? Are you just going through regular support channels? Can they deliver different instance types (not sure what the Azure parallel is), can they deliver short term capacity, etc?

I would try to push Microsoft more here. It's not like they've stopped on-boarding new customers into that region right? What happens if you create a new account in that region?

We already tried to push Microsoft, sadly have they been not very helpful. Still trying to get in contact with somebody that can actually make a difference. After all, are we also not asking for a hundred machines. Can really not imagine that they can not somehow make the resources we require available.

I'll ask again what the person above asked: do you have a TAM.

If you don't you're at a big disadvantage.

Sorry, missed that. No, we do not.

Get one asap. Your TAM is the insider and should push for you.

we tried escalating this through CASM and it did not work. The region is blocked for every quota, even a single instance.

Some tricks you can try is to switch to a different SKU. Most Azure databases have different generations of underlying compute. They may be out of just on model. Try a different one.

Similarly, just keep trying to change the size. Often it’ll go through when someone else decommissions something.

“Being multicloud from the outset” is a very silly idea for most use cases.

The way to get more from most cloud is by becoming a partner, not just a customer. And the way to do that is increase dependency and usage.

t. Sales department of Cloud provider

Good sales (which I'm not), is all about aligning with the customer: solve their problems for them, and they'd be happy to pay for it. Getting them to buy stuff they won't need is a sure way to loose future business.

> Deprioritize all other work, get everyone working on this little DevOps/Infra project.

This is doubly worthwhile as if this stumble kills the startup (it can happen) this will be excellent experience to take to the next employer :)

Also a lesson here is don’t create any prod asserts manually ever. Terraform or some other software to define your cloud assets. Then this issue is just a matter of adding a top level loop or maybe adding a region parameter to a layer of software. Cloud is only efficient if you take the software defined every thing seriously. Otherwise it is premium hosting where you are likely a small fish.

Thanks largely to k8s, running on multiple cloud providers and your own hardware is a lot more convenient than it was a few years ago. Component interfaces and protocols are a lot more consistent across platforms as well.

> everyone working on this little DevOps/Infra project.

Everyone? That's not going to help.

Reply on news.ycombinator.com