Hacker News Reader

Comment by silviogutierrez | Hacker News Reader

Very interesting article. But I have to ask: would taking down the system for a couple of hours be that bad?

I looked at the company, and while they seem rather large, they're not Netflix or AWS.

I imagine they need to be up for people to be able to check in, etc. But they could just block out the planned maintenance as check in times far in advance. I'm sure there's a million other edge cases but those can be thought out and weighed against the engineering effort.

Don't get me wrong, this is very cool.

But I wonder what the engineering cost was. I'd think easily in the hundreds of thousands of dollars.

View on HN

I think it's always worth questioning both sides. Why is downtime acceptable? People on this site routinely complain about windows needing a restart for system updates while boasting about their Linux servers uptime.

People talk about how kubernetes is overkill for many people, but it gives you rolling deployments for your applications out of the box.

There's also the "slippery slope" argument. A 0 downtime migration means 0 downtime. A 5 minute migration creeps up to a 30 minutes migration occasionally, and then regression to the mean happens, causing standard migrations to be 30 minutes.

A lot of 0-downtime migrations end up turning into a months-long ordeal and can still fail.

Allowing maintenance windows means you can do things in a much simple manner. Need to take a snapshot of a DB without running into issues with a production system adding data? Sure, go ahead, you just saved two months.

It's the old can/should argument.

The principle of something like zero downtime or switching a vendor out to save 20% may not stack up against the reality in opportunity costs, employee burnout, and customer turnover.

My company is reorganizing and so everyone is trying to settle old philosophical debates during The Churn. But everyone is overextended and a number of us have started pushing back. Just the meetings to discuss changing vendors might push the break-even point out 6 months, before you even get into implementation costs. The more time we spend working on infrastructure instead of customer-facing features and bugs, the more likely those customers are to wander off.

> Why is downtime acceptable?

At least where I'm working downtime is acceptable because our customer base is composed of professionals in north america that work 9-5 thus taking the DB offline for five minutes at midnight PST has essentially no cost to our client base. To contrast that, spending three months developing a rollout plan costs our company dearly in competitive advantage.

I agree that it's always worth examining from both sides but I also think that 0 downtime migration is both a shiny interesting problem and a point of pride for developers that can cause a company to vastly over invest in a complex technical solution to a problem they never had.

I think the slippery slope argument is almost always a fallacy. In practice you can say "we accept 99.99% availability" (which is about 1 hour every year) and budget for that. Your service might go down for unplanned reasons such as an outage but it could also go down for planned maintenance and as long as you're within that budget it seems ok.

I work for a small company and it would be devastating if our database was down for a few hours.

Unplanned? Probably. But maybe it's not as a dire as you think? Azure Active Directory (auth as a service) went down for a while, globally, and life went on. Same with GCP and occasionally AWS[1]

I'm not saying there's no downside, I'm asking against the downside of engineering cost. And that itself carries risk of failure when you go live. It's not guaranteed.

[1] AWS is so ubiquitous though, that half the internet would be affected so it makes it less individually noticeable.

So don't host on AWS, wait out their next big outage and then take down your app for that big migration, so you can hide among all the other dead apps. Half joking, of course.

That's brilliant. Have the PR ready to go and merge/deploy only when that happens. Just host on us-east1 and it'll happen eventually.

I doubt that.. aws is down sometimes too. Card payments are sometimes down, even 112(911) is sometimes down.

I hope you have everything redundant, including the CEO

The database has a hot standby and two warm standby instances, redundant hardware, network infrastructure, all of that.

> I hope you have everything redundant, including the CEO

Made me laugh, I'll sure reuse it!

OTOH I worked for several small companies that had "business hours" software - if you had to take downtime on the weekend or after hours you'd be looking at impacting <1% of users.

In my view that's just saying at some point in the future the company will be devastated. Nothing has 100% uptime.

Then hopefully everything is HA in your company.

It absolutely is, BGP with geographically diverse paths, databases, app servers, etc, are all redundant. It's hosted in-house, so there is a cold standby database in AWS which would only be used if, say, an aircraft crashed into our server rooms.

Could you share what domain your company is in? Just curious what has such HA requirements

It always amazes me when someone says they can't ever be down, and then says they only serve from one physical location actively.

One of these things is not like the other.

We have everything in place to run from AWS if needed but do not operate from there because of cost.

This goes both ways -- there have been many AWS outages which have not affected us. I hear what you're saying, but we've had only one instance of extended (hours) downtime in the last 20 years.

The more experienced I become, the more such down to earth solutions seem OMG SO MUCH MORE reasonable than all the bells and whistles of "modern" engineering practices.

Out of curiosity, what do you use in AWS to keep a running live backup of the data you'd need to migrate in the worst case scenario?

For postgres we send WAL files to a server in AWS which processes them. To bootstrap the database initially we sent zfs snapshots, and those WAL files are applied on an ongoing basis. If our data center were to die a horrendous fiery death, we could lose, at most, about 3 minutes of data although monitoring shows that it's closer to 30s under normal operating conditions.

For the app servers, saltstack is used and we synchronize that repository with what is needed to reproduce a production environment in AWS.

Obviously we'd have to provision servers, etc, but it's all possible in a worst-case scenario.

pg_upgrade with --hard-link option will upgrade the db in mins, I've done a 2TB db in less than a minute. Yes, there is no rollback, but if you do enough testing prior it can be really smooth. Am in the middle of doing this across an environment where we have tens of hosts from a few GB to couple TB.

You could take a filesystem-level snapshot for rollback.

The problem happens when you estimate it will only take a few hours and at the end of the first hour it's 1% done, or it nears completion and crashes, in a loop. Now what?

A few minutes of downtime wouldn't be that bad, but it would harm the company's prestige. Unlike AWS and Netflix, we are not based on massive consumption, but on a few customers. Losing one good customer can make a difference. We believe it worth the investment of time.

Update: The cost is a good point. It wasn't that high. We actually updated the article to reflect this. It was two engineers, working full-time on it, for three weeks.

Reply on news.ycombinator.com