Hacker News Reader

Comment by peterwwillis | Hacker News Reader

Production load balancers don't treat an HTTP response of 5xx as "success" and therefore won't continue to send traffic there. They also have periodic sanity checks of dynamic content which must match certain criteria or the host gets flagged. Monitoring systems also keep track of various periodic low/high/averages, tail access and error logs, alert on unusual criteria, and can trim the hosts in the scoreboard in extreme cases.

You'd typically learn this after probably six months of running a large-scale continuously-deployed dynamic website and it breaks from poorly-tested configuration changes + hardware issues. Sysadmins know this stuff. That's why there is a job title of sysadmin and not "developer who does sysadmin stuff sometimes".

View on HN

5xx may be the correct response - sometimes the server is asked to do something valid (ie, not a 4xx/client screwed up), but had an error when it tried it.

No load balancer I know of will remove a web server that returns a single 5xx from its healthy pool. It will need to use some heuristic as Rachel points out, some percentage that is based on the statistical norm. Otherwise it'll fail out too many hosts and cause a problem.

I think you're severely underselling developers. I've met people who have only had the Software Engineer title and never installed a Linux distribution who get this stuff at least as well as anyone who calls themself a systems administrator. Sysadmins don't have a monopoly on understanding failure cases and failure handling - false positives, false negatives, outliers and outlier detection, metrics to look at. It's a skill that comes from experience, which can happen (or not) whatever you call yourself or others call you.

I'm lucky enough that I get to focus on this type of problem, and while there's definitely an aptitude portion, it is also a teachable skill - I see my job partly as getting the team I'm working with to be able to do this stuff when I'm not around. That usually means finding two or more people in the team and cultivating their interest in it.

If your sanity check in your load balancer passes only on a 200, it will fail on a 500, disable the host, and keep retrying until it gets a 200 again. It helps for there to be more than one single request to try in your sanity check.

For "random" requests, if you have a 500 response, requests of the same "type" should not longer be sent to that host. This can be changed based on scoreboard settings. Depending on the context, you may choose to serve cached content on 500s. This is one of the reasons multiple layers of cache and application intelligence is so handy.

I'm not underselling anything. Domain-specific knowledge comes with experience. If you ask a mechanical engineer 'What's wrong with my car if it makes the noise "bang-sputz-sputz-screech-screech-screech?"', the engineer will start making you lists of what parts can make each of those noises and begin cross-referencing to see maybe in what conditions a combination of those might happen. The mechanic will immediately tell you that for your 1991 Mercury Sable, the A/F mixture is off, the MAF sensor needs cleaning, the radiator has a crack and the accessory belt needs replacing. Sysadmin is a trade, not a skill.

Okay, if we're only considering active health checks, then I'm not sure any load balancer considers a 5xx a success by default, let alone a "production" load balancer.

For a non-healthcheck 5xx response, it almost never is clear that this host in itself is responsible for the 5xx response. 5xx is the correct response when there is an error on the server side (ie, not an error on the client side, not a correct response), but it doesn't mean the server is a problem - it just means that the server experienced a problem in serving the request. That failure itself may be from one of many RPCs that server made to other services. As such, all web servers behind the load balancer for that request type will exhibit the 5xx response (at some rate, and depending on any state in connection sharing/reuse between the server and their upstream service), and all would subsequently be removed. Which isn't the correct response at all.

As someone who has had the job title "Systems Administrator" and the job title "Software Engineer", and currently has neither but still does exactly what he's always done - solving problems by understanding systems and, among other things, by writing code - I wouldn't consider load balancing and failure domains/types/handling as the sole or even primary purview of a systems administrator - especially in the case of large installations.

There's different kinds of load balancers, and as such different responses to different criteria. If you don't want to serve 500 error pages to all your users, one of your load balancers (or "proxy layers", for more or less intelligent forms of load balancers) should be doing something when you're getting 500s, like moving traffic around, or serving different content. It's far too common for 500s to be due to a machine-specific or network-specific problem to just assume they'll resolve themselves or are unresolvable.

I agree with your first paragraph, but not so much with your second. This is one reason why people pay good money for production load balancers. This is also why people run independent scripts to retrieve test pages on their production servers, and monitor in general.

This is not a solved problem, but it's also not a novel problem. It's a problem that I've faced before, and it requires decent monitoring to catch.

Reply on news.ycombinator.com