Hacker News Reader

There's actually some precedent with very high volume services taking the minimal downtime approach to avoid the increased amount of development time. Take Airbnb for example [1]. They chose to take two weeks to migrate their Messaging database with a few minutes of downtime rather than take longer to do it with no downtime. In the article, they discuss how they thought about the tradeoffs.

[1] https://medium.com/airbnb-engineering/how-we-partitioned-air...

> Having no-down-time when doing migrations is not an easy topic.

Depending on the nature of the migration and the type of data and the type of service and how coupled everything is, it's impossible. Especially when you are dealing with financial data and transactions that you can't afford to screw up even a single one.

> I usually vote for having some planned non-working-hours downtime.

Yep. No other way around it. You tell your clients weeks/months ahead of the planned downtime. They'll understand and migrate in the middle of the night over a holiday weekend.

But this is in regards to major structural migrations, not simple schema updates or changes.

Are you saying no downtime is impossible or merely expensive?

Erlang/OTP with relups is an example of a system that provides zero downtime online upgrade and rollback with zero downtime.

I'm talking about serious systematic migrations, not upgrades. In these instances, it is impossible to avoid downtime ( whether it be a few seconds, minutes or even hours ). It's one of the reasons why SLAs do not offer 100% uptimes.

Interested in a concrete example of where you see a migration being impossible without downtime; I'm not aware of Amazon or Google taking their APIs down for maintenance, which suggests that you can solve these problems with enough engineering spend.

> depending on... how coupled everything is

If you're just saying here that some systems aren't currently able to migrate in a hitless way, rather than claiming that it would never be possible for some systems, then I fully agree. Hitless migrations require a lot of work, and probably aren't worth the cost for many applications/systems.

Neither Amazon nor Google run off of an RDBMS.

Relational databases are awesome, but it's much harder to achieve 100% uptime compared to schemaless, distributed, replicated data stores. You easily can take down your system with one innocuous-looking DDL migration that works great in test but grinds the machine to a halt on a production dataset.

That's a good general point, but Youtube does use MySQL:

https://www.mysql.com/customers/view/?id=750 http://highscalability.com/youtube-architecture

Good alternative to downtime: read-only copy of application.

As with most thing, it depends, right? For one, you’d have to actually implement a read-only version of your application, including creating an understandable user experience for that scenario. That’s an additional task per feature, though, as opposed to an additional task per migration as described by the article.

Additionally, that could cause a lot of headaches for your users if you’re continuously deploying. Then again, I kind of question what you’re doing if you’re constantly running DB migrations for a user application.

Is it really much extra work to do the migration online instead of offline? It's just that second step where the application writes to both tables (and usually a step where reads switch to the new tables). Everything else is necessary for either method.

In many situations taking downtime for it is more expensive for development. When taking downtime, often you need to be really damn sure it works beforehand so you don't take unnecessary down time. Having to roll back and try again later after fixing something (and eating more downtime) can be a bit embarrassing. When you are doing migrations without downtime you can take your time between each step and add additional validations that might be impractical when trying to minimize downtime.

If you don't need to minimize downtime then maybe it doesn't really matter but that situation seems rare from my perspective.

> In many situations taking downtime for it is more expensive for development.

This is absolutely true, and the main reason for this is that the tooling sucks. Actually testing that your migration will result in your intended database state is difficult and thus people simply don't do it.

I've attempted to remedy this situation by writing a diff tool for Postgres, so you can confirm your migrations will result in a matching schema, and autogenerate more correct migration scripts.

I also talked more about this topic at last year's PostgresOpen conference: https://www.youtube.com/watch?v=xr498W8oMRo

My teams have never had any difficulty in schema changes. To be honest, we don't even think of it as an issue.

1) No deleting or alternating columns needed by any production code. Need something different? Create a new column. After weeks of this column not being needed, we use 2) to delete it - but it isn't a priority.

2) Adding/Modifying columns just use pt-schema-change. I have seen this tool transform billion row tables - it also supports slave master.

This is the same story as service dependencies, as well. If you've doubled up on this, then you can actually hide your database dependency behind a service contract. If done perfectly (high bar), then you take full ownership of the migration and nobody external to your services should have any knowledge/care of your database schema. Or your service implementation.

Nor is this where this trick ends. It is almost literally how Intel survived the supposed superior instructionsets of RISK back in the CISC days. They turned their implementation contract of CISC into essentially a service ocntract of "the computer will execute this for you." Then, took advantage of the misdirection and were able to get many of the RISK benefits in their computers.

Obviously, this can burn you at an extreme. That fun quote of another layer of abstraction, and all of that. I would be surprised if there is a prescriptive "this is how you do it" that is applicable to everyone. To note, if you only have your company as a client of either service or schema, it might be faster to just make the change in one shot.

Let that sink in. The 5 step process outlined above requires manpower to execute. Don't ignore that.

> That fun quote of another layer of abstraction...

Haven't heard, please elaborate.

Apologies, I used the wrong word. I should have said indirection. The David Wheeler quote on this page: https://en.wikipedia.org/wiki/Indirection "All problems in computer science can be solved by another level of indirection"