Hacker News Reader

Comment by mryan | Hacker News Reader

Because I would still need to run NGINX alongside it to handle HTTP proxying. By removing HAProxy from the stack, I have one less component to manage/maintain/upgrade.

I've used HAProxy for a long time and been very happy with it. But, everything else being equal, a stack with n-1 components is better than a stack of n components.

View on HN

Why not use HAProxy for HTTP proxying? If you want to drop one component from the stack it could just as well be NGINX.

Because HAproxy's ability to work with URLs and headers is close to unusable.

Surely you must have misread the docs, it's pretty damn powerful.

I ran 1.4 in production at a 8,000+ QPS social network, have been on a team who submitted patches to Tarreau that are now in HAproxy, and very intentionally put Openresty behind it for HTTP after months of tweaking a very fragile HAproxy configuration with several applications hanging off our property's domain name. I also architected and built a LBaaS product at a well-known hosting provider using HAproxy. I didn't arrive at that conclusion by misreading the documentation and I stand by it. It's also not a knock of HAproxy, it's just a reflection that being intelligent about HTTP is not HAproxy's primary use case. It is awesome for TCP and I exclusively use HAproxy to balance TCP in a protocol-agnostic way.

First, doing anything intelligent with HTTP slaughters HAproxy's performance by an order of magnitude because of the way you must configure it. Second, sticking requests to a backend is easy if you have a header that you want to decide upon. If you want to elect a different backend based upon a path component, this is much harder and yields an unwieldy configuration.

HAproxy is not designed to operate extensively on HTTP. It is designed to balance quickly and efficiently, and grew HTTP intelligence because people started wanting the convenience of making HAproxy do far more than its core focus. Rather than HAproxy getting smarter about HTTP, I'd much rather have the protocols that service my applications handle themselves and use HAproxy for its bread and butter, TCP availability and balancing. I can then focus on optimizing that using HAproxy's really clever mechanisms, like keeping the entire TCP conversation in kernel memory without reading it out (which you must do to "be powerful," as you say). This also means if I want to support SPDY or HTTP/2 or Websockets, I'm not waiting for HAproxy to support them because I painted myself in a corner.

The stack I've deployed at the frontend of every startup I've ever consulted for or operated looks like this:

          /- [haproxy AZ A] -- [openresty AZ A]
    [ELB] -- [haproxy AZ B] -- [openresty AZ B]
          \- [haproxy AZ C] -- [openresty AZ C]

This is my Standard Frontend Deployment A. My other Standard Frontend Deployment, B, is if I have the budget and comprises Netscalers because of my experience with them from Google and other companies. Startup/low budget, ELB/haproxy/openresty. High budget, Netscaler and done.

We are speaking to years of my own operational experience. I apologize if it sounded like dismissal; I actually think Tarreau would agree with my observation and opinion, if I'm perfectly honest.

I'd challenge ELB usage in such configuration. R53 should be enough.

It's nice for clean removal of a HAproxy from rotation as well as insulating against HAproxy failures without worrying about DNS caching, not to mention wildly different DNS behavior on different platforms. Some platforms unconditionally use only the first address in a RR A, which is why BIND (and maybe R53) has the "randomize A records" functionality.

ELB isn't magic either. You have to CNAME to it (i.e. mandatory AWS lookup) or use R53. Combine that with ELB slow on-ramp issues on load spikes and the fact that you pay per Gb of traffic passing through and I don't see a benefit. R53 has various advanced synthetic record types as well as healthchecks.

Wow, neat. To be honest, I had no idea Route 53 had grown that functionality; I'm only just now getting back to a startup and have been in Netscaler land for a while. In that case, you bet, I'd use Route 53 now instead. That actually sounds pretty useful and I'm going to check it out. Thanks for the tip.

I am about getting involved with a startup and I would like you to explain "Standard Frontend Deployment, B" a bit in detail. And What is ELB?

I'm not entirely disagreeing with your post, but there's a few comments I'd like to make.

1) HAproxy does support SPDY, now via NPN/ALPN. And you don't need http mode for this.

2) HTTP performance (and performance in general) is now much better. Particularly on newer kernels (IIRC somewhere in the 3.12-3.15 range, when splice was fixed for small objects).

3. I'm not sure why you found it fragile. I've run HAProxy in HTTP mode at much higher QPS volumes than you were seeing - this did require a bunch of tuning, and I'm actually hoping to have time sometime in the next few weeks to write some articles on doing this. But it worked, and worked well.

What's worked well for me is the following:

           /-  [haproxy]
  [router] --  [haproxy]
           \-  [haproxy]

Using anycast routing to distribute the load over the haproxy servers, who run BIRD[1]. It's simple and pretty effective - although you want to chose your router hashing method carefully, and if you use persistance (stick tables) you have to use a recent version of haproxy, with support for peering.

It is difficult to make it work, though - just how difficult I only recently appreciated when trying to help a friend over IRC make haproxy scale to very high volumes. I'm hoping that I'll find the time to write the articles I mentioned above, which will hopefully be useful to others with this sort of problem to solve.

1: http://bird.network.cz/

Well, I don't know what you refer to by "doing anything intelligent with HTTP slaughters HAproxy's performance by an order of magnitude". What particularly intelligent makes it slow ? I'd say it's supposed to be quite the opposite as we take great care of making it possible to write almost any configuration without having to use regex for example. We've added rules to rewrite parts of the request by combining other elements thanks to the format strings being used in all HTTP rules. You have sample fetch functions which extract various contents quite quickly and allow you to reinject them anywhere very quickly as well.

Some users reported more than 400k requests per second in HTTP mode, or 50 times more than what you experience. Sure, here any form of HTTP processing adds a few nanoseconds to the processing time and will slightly lower the numbers. But 8k is the level of performance you should expect from tens of thousands of HTTP rules which probably is not what you're doing.

So I'm interested in knowing what trouble you're experiencing. Feel free to bring that to the mailing list, a design is always better when more people are involved.

nginx can do a lot more than just HTTP proxying - it's a pretty popular webserver, and if your stack contains both nginx and haproxy right now, you must be using nginx for something haproxy can't do. So if you use both, you can drop haproxy. If you just need a reverse proxy, there's no reason to replace haproxy with nginx.

Reply on news.ycombinator.com