> Consider hot code reloading to be like changing the tyres on a moving car, the pieces and tools are there to make it possible, but in almost every situation it's more sensible to stop and change. Backwards compatibility for all API changes, code correctness and deployment methodology need to be fixed before you can start a hot rollout. That's not even to consider a flexible application architecture that will be compatible with such changes in functionality (OTP does indeed ensure this is compatible on the basic process level, but there is no magic bullet around high level architecture).

This was an interesting discussion I had with a few of our B2B customers at a customer event.

Even with redundancy and everything, something like a major postgres upgrade is a tricky thing. You either have to do some amount of work of setting up several clusters, setup logical replication, decide a cutover point (most of them things pg_easy_replicate (if I remember the name right) does), while juggling the backup solution on the side and such. It's overall understandable, but a decent number of steps to do right, and if you don't you might even lose transactions to the upgrade. And losing transactions is way worse than being down.

The alternative though is somewhat brute: After testing everything, shut the app down, shut the cluster down, upgrade binaries, run pg_upgrade. Restart cluster, check cluster, start application. This requires downtime, but as a procedure, it is very straightforward, very testable and overall predictable. Running a readonly version of the application sadly wouldn't be possible because of choices in the application.

Interestingly, many of our larger customers obviously preferred if vendors don't go down, naturally. But for a complex migration or update, most of them have been burnt by horror stories of upgrades going entirely sideways. If an outage is of predictable time, well-planned and communicated in advance, they can prepare for it and deal with it. One just shrugged and said "So what, then we just tell our customers we don't provide that service for half a day. No big deal with a month in advance or so"

This gave me a very valuable perspective of how much energy to put into avoiding downtimes for rare procedures and interactions. Talk with your customers, and sometimes the correct answer is indeed "don't".