Hacker News Reader

Comment by valenterry | Hacker News Reader

It depends. If you want a service that doesn't have downtime because of a hardware issue, then you should have at least 2 machines. And then, you probably want to do deployments in a way where you spin up 2 new machines and then, only once confirmed that the deployment works, shut down the old 2.

That is super easy to do on AWS (and many other providers) but not so easy (or even impossible) anymore when you do things on-prem.

EDIT: due to the replies, let me clarify; this is not an argument for AWS specifically but for services that make it easy to spin up machines and that handle deployments and failover for you.

Also, there are more reasons to have more than one machine, e.g. being able to deploy with zero downtime and therefore without having to think when to deploy and what to do if a deployment fails etc.

View on HN

Most web projects can tolerate an hour of scheduled downtime a week and the occasional (< 1/year) unscheduled hour of downtime to replace broken hardware. Running on a single machine with 2 disks in RAID1 right up until the point where the ~50 hours of downtime a year equal one engineer's salary in lost revenue makes sense if you pick a DC that will sell/rent you replacement hardware and do replacements for you. Just make sure you have offsite backups in case the DC burns down. If your office/home is near a well stocked computer store, most web projects can skip the DC altogether and just put the server in the closet under the stairs or whatever. The company will likely die before the server does anyways.

> Most web projects can tolerate an hour of scheduled downtime a week and the occasional (< 1/year) unscheduled hour of downtime to replace broken hardware.

An hour? What happens if it breaks while I'm on vacation and no one is there to take care?

Nah, that argument might work sometimes but not in general.

Yes, an hour. In practice, most scheduled maintenance will take closer to a minute than the full scheduled hour. If it's good enough for the government and the bank, it's probably good enough for you.

What happens if you're on vacation and something breaks is that you call/email the DC and ask them to replace what's broken, same as if you were not on vacation. In my experience most DCs will have the replacement done in less than 30 min from receiving the ticket, good ones will do it in under 15. If the project is ran by more than one person, simply don't all take a vacation at the same time. Worst case scenario you point DNS to a free static page host that reads "Server broke. Working on a fix, check back tomorrow." (or the appropriate corporate-PR phrasing of the same if applicable) while you wait for the computer store to open so you can buy a new one.

Even if you are once-in-a-lifetime unlucky and the damn thing breaks while you're on vacation and for some reason your monitoring fails to notify you, big whoop. You can probably get away with just apologizing for the inconvenience and moving on. If a week long outage is enough to make or break your business, your business isn't gonna work out anyway.

This applies to practically all web projects from the home page of the pizza place on the corner of your street to 100-employee e-commerce sites. If a single machine can run it, it can probably run on a single machine. Use the dumbest tool that will get the job done, switch to a smarter tool when it won't.

Even just setting up and documenting all the things you mentioned is already taking so much time and effort, that I'd rather pay a few more bucks for a HA system...

> If a week long outage is enough to make or break your business, your business isn't gonna work out anyway.

If your business has an SLA with its customers, having a week long outage is going to break your business. Some contracts I saw had a 15 minute reaction time where you had to acknowledge the issue, and after 1 hour of downtime there are penalties. The same contract also required redundancy (of hardware, internet lines as well as location for the DCs). We could not have hosted that on a Raspi, or a single machine.

I don't think reducing the scale issue to "you ain't gonna need it" is right. Obviously to most projects it's not gonna be needed. But when it does and you need to grow, you'll be happy you have a redundant infrastructure that you can spin up/down easily instead of having to call your DC to ask for more hardware to be installed.

To be fair, that is probably out of scope for what OP meant.

Maybe for the author of the blog post, about his personal projects. But the user I replied to seemed to indicate even "100-employee e-commerce sites" don't need any redundancy for their website, which I disagree with.

"If your business has an SLA with its customers" then you also need a paid SLA with Amazon and now we're talking about the 100k annually cost range where you can also afford to hire freelancers to babysit your sever while you're gone.

What happens if AWS has a hiccup that restarts your VM?

The answer might be to buy management services. Or an SLA for AWS.

Most but not all.

If you're running an online shop/reservation system, a hour of downtime can be tens of thousands of lost revenue.

Revenue, not profit. Many small shops voluntarily turn off their website for some weeks so that the employees can take a Christmas break. So it seems like the business can handle a week offline.

Super easy? Getting that setup for the first time from zero knowledge will probably take a few days. And that's before understanding all the intricacies like your AWS bill, the hidden costs of cloud, properly securing IAM and setting up your VPC etc etc

Not once in my entire career have I seen people successfully avail themselves of the cloud's purported benefits. I have, however, seen a lot of happy account managers in Las Vegas during Re:Invent, oh yes.

I got downvoted for saying something similar for standing up a web app from scratch and getting it deployed securely.

You are absolutely correct

Comparably easy, yes. I'm not talking specifically about AWS btw, but even there it is easy.

If someone has zero knowledge, then everything will take them a long time, including hosting on-prem or so.

It is exactly as easy, or easier, to do this in proxmox than it is in AWS console, and would have a price breakeven point of less than a year (you only need two physical nodes).

Proxmox is just a way to virtualize things. You need hardware to run it. So what hardware are you talking about?

Missing the point that you don't need to be able to spin up/down servers. You can just rent 2 and spin up/down containers.

> You can just rent 2 and spin up/down containers.

Missing what point? You are apparently agreeing with me and disagreeing with the OP who said: "can run on a raspberry pi or mini PC".

No, you most definitely can just spin up/down containers on a raspberry pi or mini PC as well.

This is what I objected to:

> And then, you probably want to do deployments in a way where you spin up 2 new machines and then, only once confirmed that the deployment works, shut down the old 2.

You don't need to spin up more machines or shut down machines. As long as you ensure you have the headroom you spin up containers and shut down containers. By doing that, it's also relatively easy to set up hybrid setups where you can handle load-spikes by scaling into EC2 or similar if you should ever need to.

> As long as you ensure you have the headroom you spin up containers and shut down containers.

Are you very sure you are not introducing potential additional problems with that approach? Because if you do, then I wouldn't say it's "not needed" to spin more additional machines - it would depend on those additional problems.

Yes, I am, very much sure of that, based on 20 years of experience deploying containers that way, and nearly the same length of experience using cloud instances. They are equivalent in complexity. It is purely down to how you want to host things and whether cost matters to you - the managed approach is consistently significantly cheaper for all but the very, very smallest environments where factors like discounts and free tiers play in.

Literally any random x86 box you have lying around. NUCs work fine here, but so do used Optiplexes from Craigslist or your old gaming PC.

Does proxmox have 24/7 enterprise support yet?

> not so easy (or even impossible) on-prem

It’s as if an entire generation of tech workers are utterly unaware that these have been solved problems for decades.

You put two machines behind HAProxy with keepalived. If you want to do a new deployment, you can spin up new machines in the same manner, and cut them over (or any of the many other ways to accomplish this).

AWS isn’t inventing any of the HA stuff they offer; they’re taking existing technologies and packaging them.

So how do you do all of that with "a raspberry pi or mini PC"? You would need at least 4 of them.

I was replying to your comment, not the other. But sure, get 4 RPis if you want. You could also be using containers on them, and not need to swap whole computers. Or run each app in its own chroot jail. Or run VMs. There are many, many ways to accomplish this goal.

Most web projects can afford to be down for a bit. Even the big guys have outages fairly often. It's not like AWS is always running smoothly either, and there are so many "_ is down" posts on HN alone.

I do agree that your projects should be designed to be well contained and easy to create a new instance of it, though.

> That is super easy to do on AWS (and many other providers) but not so easy (or even impossible) anymore when you do things on-prem.

False. You can run kubernetes and/or some virtualisation engine as well and do the same on prem.

and overprovisioning physical hardware is much cheaper.

Well, yes. I would categorize "running kubernetes on prem" as "not so easy".

And you can easily have a separate clean environment for every pull request easily.

Can't do that with your own hardware unless it's also a Kubernetes cluster, which is a whole new set of crap you need to manage.

I'm sorry, but the ten billionth CRUD web app really doesn't need nine nines.

Just a blue green! But I agree somewhat. Single machine running Docker compose is pretty decent. Pets! Not cattle. (I'd rather care for a dog than a 100 sheep)

I agree. It doesn't change my argument though.

Digital Ocean I believe is pay as you go and supports this for anyone who can click "link my Github account" and auth themselves.

Digital Ocean is not equal to "run on a raspberry pi or mini PC and be just fine".

Reply on news.ycombinator.com