Ask HN: Why do websites have scheduled downtime if AWS/GCP prove its not needed? | Hacker News Reader

Ask HN: Why do websites have scheduled downtime if AWS/GCP prove its not needed?

Something I've always wondered about

Big cloud platforms never take the service down for maintenance

They do any upgrades or maintenance live on the running service

If this is possible for them, why do so many websites have scheduled maintenance where they take the whole site down?

Are the engineers just not able to figure out how to do the maintenance without turning off the website?

What is the secret that these cloud platforms have discovered where they don't need scheduled maintenance like the rest of the web?

View on HN

If the military has fighters that can refuel in the air, why does United refuel on the ground like chumps?

First of all, sometimes even AWS makes mistakes and has downtime. They have downtime (usually unplanned) fairly often. Companies like VISA, that do payment processing, where countless dollars could be lost in just a few minutes of downtime, almost never update their old systems. They still have programs running FORTRAN. When they do update, it's complicated.

You can try to get really good uptime by doing things like forking your database and replicating your commit log to the fork. You can do rolling updates, where different customers are served different versions of a system. You can do extremely thorough testing.

But are you? Can you afford to do that and do you even have the engineering talent to make it happen even if you want to? Are you going to pass on new business opportunities and new features just to avoid a 20 minute downtime for a database migration at 2am? I suspect you are not.

You can stretch availability to 99.9999999 if you have the money and the team to support it. However, each jump can incur up to a 100x cost increase per the Google SRE book. Availably is always a conversation about risk, innovation, cost, and what your customers will accept as an SLA.

Even Google doesn't go 99.999 on all services. Services that require rapid innovation are always lower because availability incurs an opportunity cost on innovation.

I'm not talking about general availability, just wondering why would a whole website need to be shut down to do maintenance?

What is this specific kind of maintenance that cannot be done live?

Same difference... read the book

https://sre.google/

It generally takes more prep work to do maintenance without downtime (i.e. it’s more expensive), and if you’re not operating at their scale, it’s less likely to be worth it.

Because most sites don’t want to spin up 100000 instances in 40 seperate data centers.

Why would you need 100K instances

If you run at web scale and serve ~10 million customers, you'll need a lot of servers to keep running. Most sites aren't anywhere near that.

Ya but say I'm a small business and only have 6 servers for my website

Why do I need to take all 6 offline to do maintenance?

Because you (accidentally or otherwise) built your infrastructure to need to. It's possible to build it so you don't have to, but most don't.

Because likely there is one master, authoritative data source and that has to go down at some point.

If you need to change something in your DB wouldn't you just have a job running in the background doing the migration and have your application code handle both cases when reading the DB (migrated/unmigrated)?

You say "just" like you think that sort of thing is easy. Nothing could be farther from the truth.

For one thing in many DBs certain (often nessesary) schema changes require a global write lock.

It's much much easier to just show down for 15 minutes or less than spend hundreds of hours of engineering effort.

10 million isn’t web scale though. Quite a few cities, many states and a bunch of smaller EU countries would have most of their b2c businesses, local government services and heck, even local newspapers serve 10 million people. That’s not web scale, that could be SQLite on one machine.

Tyler's point is that most companies don't have anywhere near the computing scale and workforce size which Amazon and Google have. Most companies don't have any option other than to take services offline during certain types of maintenance.

And why _exactly_ is that?

Why does the service need to be taken offline?

Because more instances costs more money and are more complex to orchestrate, costing even more money.

Most sites don't have the SLA or other reliability requirements that dictate crazy uptime. If they have no need, they won't structure their infrastructure for it because doing so is expensive. Big cloud companies have that figured out because their customers want it. Other, smaller sites feel that their money is better spent elsewhere.

Reply on news.ycombinator.com