Hacker News Reader

Comment by skullone | Hacker News Reader

So they used a feature built into a load balancer to gracefully drain traffic from specific availability zones? Odd that a feature found in load balancers from the last 25 years is a blog post worthy thing.

View on HN

Close but I don't think it's quite 25 years! I added graceful draining to Apache httpd's mod_proxy and mod_proxy_balancer either in 2003 or 2004, and at the time I'm nearly certain it was the first software load balancer to have the feature, and it wasn't available on the hardware load balancers of the time that I had access to ... though I later learned that at least BigIP load balancers had the feature.

At the time, we had healthy debates about whether the feature was useful enough to justify additional complexity, and whether there could be cases where it would backfire. To this day, it's an underused feature. I still regularly run into customers and configurations that cause unnecessary blips to their end-users, so it's nice to see when people dig in and make sure that the next level of networking is working as well as it can.

Well played, HN.

I migrated some old BigIP load balancers over to Apache in 2004ish, and extended some of mod_proxy to do some "unholy" things at the time. We also did a lot of direct server return stuff when no load balancer you could buy could handle the amount of traffic statefully. Man, how times have changed, and lesson forgotten.

Microsoft bought Convoy in 1998[0]. Then incorporated it into NT4sp6a and Win2k as NLB/WLBS. One of its features was to gracefully remove a server from the cluster after all connections were closed - draining. But, cluster not the same as an LB.

[0] https://news.microsoft.com/1998/08/24/microsoft-corp-acquire...

That seems like a shallow dismissal. In a distributed system, making sure that sub requests are handled across distributed nodes within the local AZ, and correctly draining traffic from AZs with partial component service outages, is not as trivial as 'using a feature built in to a load balancer'.

It may be shallow, but architecting for this is not really "advanced, FAANG-only accessible methodology". I'm surprised their services have been as "reliable" as they have been considering such trivial stuff is just now being employed in their architecture.

> architecting for this is not really "advanced, FAANG-only accessible methodology"

Sorry - where are you quoting this claim from?

The S in FAANG is for Slack.

My own words, but this is fairly trivial in the context of these massive companies with presumably PHDs working on their architecture.

I don't quite know what a PhD-grade architecture would look like, but this seems like a reasonabke one to me.

Half the complaints on here on architecture posts are 'you don't need this kind of stuff unless you're at FAANG scale'. Now we have a write up of something that's accessible to businesses at non-FAAANG scale, and we have the new complaint, that this kind of stuff isn't worthy of FAANG-scale architecture.

Geo traffic distribution, multi regions/AZs with functionality to weight and drain traffic should be used in most SaaS services where a simple failure somewhere could cost users time and lose company money/goodwill. It's not terribly hard nor expensive.

Those are all much looser restrictions than routing traffic consistently to a cell

Route 53 latency based routing -> APIGW or ALB -> Lambda or Step Functions -> DDB Global Table.

No reserved capacity (pay for usage), so it works for boot strapping startups and provides superior resilience while being extremely simple to setup and involves almost zero maintenance or patching (even under the hug of death). I don’t understand settling for less (and taking longer and paying more for it).

How do you think people are going to learn this stuff if not by reading about it from architects who have done it?

This writeup seems like a useful contribution to spreading this knowledge that you think every engineer should, somehow, innately be born with, to those members of the development community who missed out on picking this stuff up in elementary school.

The other bit is separating the service into isolated cells so issues in one don't affect dependent services everywhere like they had experienced before.

But yeah any good SRE could point this out years ago.

Just odd a company worth billions and billions of dollars is just now discovering HA models standard since the 90s. Can expand the Clos network architecture to these distributed service applications too. But judging by Slack's client quality, mature concepts such as those must be new to them.

The linked AWS article specifically explains that it’s not just the typical single load balancer for cross AZ routing. I frankly don’t know where you’re getting that this means that HA is new to them.

Of course this isn't a typical single load balancer for cross AZ - but the general gist of their "new" architecture is first principles level of design. But sure, we can celebrate their minor achievement I guess

Reply on news.ycombinator.com