Scaling Down Gracefully with AWS Lambda and HAProxy | Hacker News Reader

Scaling Down Gracefully with AWS Lambda and HAProxy

contentful.com

65 pointsrealityking14 comments

What are the conditions that using HAProxy is preferable over using ELB or in conjunction with ?

When you want more control over your load balancing, or you have long lived connections that exceed the connection timeout you can configure with an ELB (RabbitMQ connections, for example; ask if you want deets, we've tried using ELBs multiple times with Rabbit, and always come back to HAproxy).

We had issues with internal ELB's for RabbitMQ (the 1 hour idle connection timeout) and the solution ended up being a heartbeat in the clients. Most client libraries have a parameter that will perform a heartbeat at some interval to keep the connection open.

I can't imagine deploying our own HAProxy infrastructure. Sure, it's relatively simple, but the set-it-and-forget-it comfort provided by ELB (not to mention free SSL) is hard to beat.

We've seen some really weird ELB behavior in AWS (latency spiking excessively every few hours for seconds at a time, underlying ELB instances having scale down events occur at unexpected times, and so forth), so we use it sparingly for internal microservices.

Running your own HAProxy infrastructure isn't terribly time consuming, but I do devops/infrastructure all day long; I'm biased. If you can and do avoid it, I can understand why.

Haproxy also supports mutual SSL and can do content-based health checks. Logs don't have to go through an S3 bucket so you can also get real-time metrics and logging if you want.

Also, because you have control over it, you have more options for service discovery (i.e. write a Python script that periodically queries a Kubernetes cluster).

And virtual hosts... And SNI... And about a thousand other features.

Honestly, ELBs are pretty crappy.

ELBs are awesome if you need stupid, simple load balancing, and aren't too picky about how they behave. Anything else? Haproxy.

I love that I can specify existing IAM SSL certs to an ELB; I don't have to provide access to the actual certs/keys to folks who want to provision their own ELBs in our environment, they simply need to pass through the IAM resource in the config. Its a small thing, but still important.

EDIT: Something to note though, from my trails of broken glass I've walked; the process limits haproxy displays in the web interface don't always match the process limits the process actually has (as determined via /proc/<process_id>/limits). Don't get burned like I did (too few file descriptors despite haproxy indicating the contrary, epic connection failures).

So HA only updates when chef runs (every 15 mins or so)? And when nodes are deleted, I guess something else has to clean them from chef? You'll also have to add these hosts to monitoring config.

I implemented something similar with puppet/mcollective/SQS years back. Poor man's service discovery. Here are some other options that, if you can handle the complexity of running Zookeper (or similar), are probably a bit cleaner:

http://jasonwilder.com/blog/2014/02/04/service-discovery-in-...

Interesting article. While my company's cloud infrastructure is in its final nascent stages, I will definitely share this article and the ideas therein for basically reloading the HAProxy context w.r.t the proxied backends.

How do you ensure that no requests are lost when you reload HAProxy?

Soft reload takes care of that for you, sockets aren't dropped.

If you restart, however, everything's dropped.

Is that new in 1.6? As of 1.5.11 there was a lot of hacking required to get a true zero downtime reload: http://engineeringblog.yelp.com/2015/04/true-zero-downtime-h...

No, he's referring to the soft reload added in 1.5, which the article you linked addresses under "Just Reload HAProxy." As the article points out, it's not quite zero-downtime, but pretty close.

Why not drop servers in a way described here? http://serverfault.com/questions/249316/how-can-i-remove-bal...

You can down servers or backends over the stats socket, but you can't add them (adding servers requires a reload).

The DNS resolution work that is going into 1.6 releases may lead to a workable approach where you over allocate servers in a backend and then update them to have the right host/port. Last I checked, however, you still couldn't set the port dynamically.

Reply on news.ycombinator.com