Hacker News Reader

Comment by z3t4 | Hacker News Reader

Syntax errors are the easiest errors to fix and almost never get into production.

Zero downtime deployments are very rare, and ads a lot of complexity, so it's a big trade-off, compared to just uploading the new code and restarting the service. And having the front-end deal with minor hiccups like service restarts, because users will have such issues all the time due to being on mobile networks, trains going through tunnels etc.

You can do as much testing as you want, both manually and automatically, but you still can't detect all issues as efficient as thousands of users in production. So just accept that there will be issues, and instead design your pipeline so that those issues can be fixed within minutes.

View on HN

> Zero downtime deployments are very rare

What do you mean? Rolling deployments are a very basic feature of container orchestrators.

You don't even need containers--it's a feature of load balancers containers rely on. Start new container, wait for it to become healthy, add it to the load balancer, remove old container (with an optional, let connections to old container close first)

what if it involves db updates?

Where I work we had that same thought. Until we stopped doing breaking database changes.

We have run a lot of db changes. It requires multiple steps. Add a column, migrate data, update code to use new column and old column, update code to use only the new column, update table to drop old column. Similar for migrating to new dbs, tables, etc. It is rare for us to need to disable writes for a version bump or something.

If you aim for zero downtime there will be a point where the complexity work against you. for example: load balancer goes down, corrupted state due to two different versions running simultaneous, a bottleneck causing congestion, etc.

My favorite "load balancing" for web-pages is using "DNS round robin" where you add several A records, and the browser automatically try another IP if the request fails.

> ...the browser automatically try another IP if the request fails.

The browser tries another IP if it cannot connect to the first one.

If a server throws 500s left and right DSN RR won't help, whereas a load balancer would remove the offending server from the pool.

DNS round robin provides redundancy, not load balancing (though it ipso facto does somewhat evenly distribute requests). You really need an application aware load balancer that can detect http status codes and route around failures at the very least.

Zero downtime deployments is not the same as zero downtime generally ever.

We deploy to production several times a day. We have an in-house product that our own emplyees use and they absolutely love that they can ask a feature in the morning and get it in a few hours (in best case scenario). That wouldn't be at all feasible if all those deployments caused errors to users. Luckily that's not a problem, because these days it's really easy to get zero downtime deployments with all this containers and "serverless" functions and whatnot.

The browser does not try another IP if the request fails; it tries the first one only (the DNS server chooses a random order for the response), unless browsers have changed their behavior recently.

DNS round robin is not failover. If one of your four A records points to a dead box, 25% of your customers are SOL.

Its supported in IE, Chrome and Firefox, havent tried Safari and others. They will wait until timeout, then try the next IP. IE will give up if the second IP is down, while Firefox and Chrome will try many IPs

https://en.wikipedia.org/wiki/Happy_Eyeballs

Browsers even try other IPs and alternate between IPv4 and IPv6 before the connection is established.

I know all about this, but most websites only support v4, and I don’t think this is used in any way for choosing which A record to use.

Most websites don't have redundant servers but its still there and used by a lot of big sites.

The whole point of it is to connect to multiple A and AAAA records with a very short delay (250ms in chrome). The browsers use poll to get notified about the first established connection and then use it. So if one A/AAAA records IP is unreachable or slow it will use a another one. There will be no fallout (except maybe for clients that don't use something like happy eyeballs, but any browser does that) if you have multiple A/AAAA records and one of them is down.

I don't know what he refers to. But there is a specific kind of zero downtime deployments which can restart without losing in-memory state, like Erlang.

Erlang also provides "fix in minutes, in production" stories he refers. For most of languages, it's very hard to remotely get access to an object, let alone fix things directly.

With Erlang, you can inspect every object in the cluster to understand what's going on for any user, and fix the plane in the mid-air instead of rebuilding a plane with the same issue, then go through the process to fix it.

> You can do as much testing as you want

With what Dark is promising, I believe testing would be easier. Deploy to a test environment in 50ms. Then run a suite of tests, then deploy to production in 50ms.

There could be some complementary tech that figures out what tests need to be run based on what you change to get the time down.

They're kind of lying.

You only need pip or any package install if your testing image doesn't pre-install the required packages before running the tests. Which I don't myself since I always want to see if my packages work with the latest versions of other packages - and Dark wouldn't be able to handle that anyway.

So my test runs actually complete in about 50 ms if I'm not testing any external API functionality. They will also never finish earlier than 5 minutes if I am, even on Dark - since they're based on external services.

Reply on news.ycombinator.com