HAProxy 2.4 | Hacker News Reader

HAProxy 2.4

haproxy.com

Author of the blog post here, happy to answer any questions about this release.

Open tracing sounds nice, what info is passed along?

My question is what conversation has there been re: Opentelemetry, the successor to OT

It sounds like ot is still in beta (but it's a little confusing that they're 1.0 but it has a message saying they're in beta on their main page).

It looks like it's being adopted by a few larger app monitoring products, wondering if datadog will follow suit or if they will stick with open tracing and their custom implementations for logging and metrics. I assume they will support ingestion from ot at some point though. It sounds neat tho, will definitely read up on it more.

You can pass along timing data from every aspect of the request [1] currently there are approximately 20 events supported. You can also send over just about anything that can be stored within an HAProxy variable / header through a tag. Here's an example configuration for HAProxy (simple) [2] (advanced) [3] and the OpenTracing specific configuration (simple) [4] (advanced) [5]. I've included both simple and advanced configurations.

[1] https://github.com/haproxy/haproxy/blob/master/addons/ot/REA...

[2] https://github.com/haproxy/haproxy/blob/master/addons/ot/tes...

[3] https://github.com/haproxy/haproxy/blob/master/addons/ot/tes...

[4] https://github.com/haproxy/haproxy/blob/master/addons/ot/tes...

[5] https://github.com/haproxy/haproxy/blob/master/addons/ot/tes...

Yea, it's pretty flexible and you can attach any additional metadata that you want (e.g. app version info, user name).

The added support for FIX is interesting. As far as I can tell it's just TCP-level load balancing, is that right? HAProxy doesn't have the ability to, say, distribute each incoming order to a different backend server?

It has the ability to route by FIX tags. You can find an example here [1]

[1] https://www.haproxy.com/blog/haproxy-enterprise-2-3-and-hapr...

Sure, I saw that example. Maybe let me put it a different way: if a client has one inbound FIX connection to HAProxy, can HAProxy connect to N backend servers and send each FIX message from the single client connection to a different backend?

I have not tested that use case but I do not think so. It has the ability to parse FIX tag values and make routing decisions from the initial Logon message.

It seems like it could:

  use_backend fix_servers_a if { var(txn.sendercompid) -m str firmA }
    default_backend fix_servers_b

Then "fix_servers_a" could be defined as a round-robin backend.

Not specific to this release, but for an system architect that is taking the first look ever to the product:

- the site does not do a good job of describing the products and what they do. https://www.haproxy.com/products/haproxy-enterprise-edition/ has a download button as the start and tells nothing about the product.

- it is hard to find requirements from the first page. (https://www.haproxy.com/documentation/hapee/2-2r1/getting-st...)

- is there any a page with basic architecture of the product? Some diagrams with typical configurations.

- is there a simpler HA setup than the 3 layer one here: https://www.haproxy.com/documentation/hapee/2-2r1/high-avail... ? What is the simplest HA configuration for 2 web servers?

Thanks for the feedback. I've provided a few replies below but I think you have made some great points and I will share them internally.

> - the site does not do a good job of describing the products and what they do.

The page does lead off by telling you it is a software load balancer, but I agree, we can do a better job of giving more information on the product itself. Most people who pursue HAProxy Enterprise are already familiar with HAProxy. With that said, we're actually working on a complete overhaul to this page and I believe the new version will address this.

> - it is hard to find requirements from the first page.

This is just a reference point and typically organizations are working closely with a sales engineer.

> - is there a simpler HA setup than the 3 layer one here:

A simpler version is found under active/standby here [1]

[1] https://www.haproxy.com/documentation/hapee/2-2r1/high-avail...

Thank you. Just for info, my personal experience working for a very big American company is that every time I call a product company for information, I get intensely called back months in a row even if the product does not meet our needs, so I prefer not to do that anymore and to dig myself for info until I figure if that is looking very promising. Maybe it is not the same in your case, but who acts assuming it's an exception?

How did you get a system architect title while somehow never looking at haproxy in the last.... 15 years?!

Downvote me all you want but “architects” need to look at the solution landscape at least once in 15 years.

Plenty areas of software where HAProxy is not part of the relevant "solution landscape".

Systems architect?! I want a load balancer. What’s your top 5 excluding HAProxy?

You don't need 5, especially when the load balancing is part of the network or infrastructure team that are a different group. Some companies have hardware load balancers, some use products like VMWare NSX or both, depending on the needs. When you are a Windows shop, HAProxy is not on the short list, hearing about it and actually knowing it are 2 different sides of the coin.

As someone who isn’t familiar with this space - is the use of HAProxy as a self hosted DDOS defense a “supported” (prioritized?) use case? I found old blog posts on this application of HAProxy but wasn’t sure if new development tries to improve those capabilities or if it is more focused on load balancing or other applications.

If I have a cloud provider, why might I use HAProxy over the provider's load balancing solution?

More control, maybe less cost? If your cloudy provider has an awful solution like the one I was using, you'd get better uptime by doing it yourself.

I haven't heard many problems with AWS or Google load balancing, but sometimes there's issues where people talk about 'pre-warming'... more control would probably help there.

If you're running HAProxy on AWS, be sure your AWS firewall rules are stateless, otherwise you may well run into the unpublished connection tracking limits.

> If you're running HAProxy on AWS, be sure your AWS firewall rules are stateless, otherwise you may well run into the unpublished connection tracking limits.

This.

I got bit by this the other day, a bunch of servers were having very low CPU usage but new connections to them were failing. Turns out it was because they were hitting the conntrack limit. Changing the security group rules to allow all from all "fixed" it.

Although after a quick test, I think there's another limit, also unpublished, as I still can't establish as many connections as I want on a smaller instance – same number of requests on fewer connections works, and also a bigger instance works with many connections. In every case, the CPU was largely unused.

AWS doc regarding the issue: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/security...

> Although after a quick test, I think there's another limit, also unpublished, as I still can't establish as many connections as I want on a smaller instance.

Any chance this is an OS limit in your instance? Either a connection tracking firewall there, or FD/socket count limits? I don't have experience beyond the AWS connection tracking limits because I ran into them while testing something, but only found out how to avoid them much later (the test didn't go well, because of the limits, so we didn't go forward, and our assigned rep didn't tell us about the workaround either; I only found out by complaining about the limit to enough people that someone knew how to fix it)

I doubt that, because the same exact VM, after changing to a beefier instance type (c5.4xl, instead of the original t3.micro and later c5.large) accepted the connections without any problem.

If it's OS related, it may be some dynamic limit based on RAM size. But the limit was something around 100-200 connections, so nothing particularly crazy.

The way I found out about the limit was that downstream servers couldn't establish connections, and a manual test showed significant packet loss, while the servers were under 10% CPU and accepted connections locally. I opened a support ticket and the guy apparently saw that they were hitting the conntrack limit.

Once I replaced the security group with a blanket allow all as he suggested, connections could be established again. But two days later, when the number of connections increased further (while the security group was still allow all), I started seeing again connections not being established, although the servers were still not particularly loaded. As these are UDP (DNS) servers, I can't usefully quantify the number of different clients.

I set up a quick lab with two isolated EC2 instances and figured there must be another limit to the number of connections, although it seems somewhat higher – or maybe the conntrack limit as shown by ethtool doesn't update immediately. This happened yesterday, Saturday, while I was on call, so I didn't push the tests far enough to reach any clear-cut conclusion. I just doubled the servers and everything went back to normal.

However, the limit does seem to be related to the instance size and per interface. Meaning that adding another interface doubled the connections I could establish. Again, I didn't push the tests any further, though I intend to this sometime next week.

One potential use case: proxying targets with no fixed IP addresses for services which require fixed IP addresses. For example, exposing RDS databases via privatelink. It would be nice if AWS would address this natively, but HAProxy will do.

You'd have more specific than "every single cloud provider" but in general HAProxy is way more advanced than most other proxies.

Stickiness, rate limiting, gzip compression, configurable logging, caching, Lua scripting, load balancing to domain names, more options for health lifecycle. It even has a control socket to instantly manage the server without restart. HAProxy is actually the inventor the PROXY protocol which a number of other systems have now copied.

The convenience of cloud proxies is no small matter, but on a feature-by-feature basis HAProxy easily dominates.

Well said, to add to this, HAProxy is also closing the gap on convenience with its recent addition of AWS EC2 service discovery [1] and Consul service discovery integrations [2].

[1] https://www.haproxy.com/blog/announcing-haproxy-data-plane-a...

[2] https://www.haproxy.com/blog/announcing-haproxy-data-plane-a...

Wow, cool. HAProxy is coming a long way. Consul integration makes me excited to take another look at using it for our mesh. Using Fabio right now, and it's fine, but I'd love to go back to HAProxy for this if it makes sense.

disclaimer: I work at HAProxy Technologies

More control/flexibility (i.e. cloud provider usually gives you a limited number of "checkboxes" that they want you to be able to modify), generally better observability (HAProxy timing metrics [1] and session state at disconnection codes [2] are very helpful for debugging), and granular control over timeouts [3]. You also get builtin rate limiting [4], which from what I have seen with most cloud load balancers usually requires an up-charge through their "WAF" product.

Better health checking, circuit breaking, and layer4/layer7 retries [5].

Another consideration is whether you have spiky traffic patterns. For AWS load balancers at least, they need time to "warm up" to larger traffic levels. If you manage your own load balancing tier you can scale on demand.

[1] https://www.haproxy.com/documentation/hapee/latest/onepage/#...

[2] https://www.haproxy.com/documentation/hapee/latest/onepage/#...

[3] https://www.haproxy.com/documentation/hapee/latest/onepage/#...

[4] https://www.haproxy.com/blog/four-examples-of-haproxy-rate-l...

[5] https://www.haproxy.com/blog/haproxy-layer-7-retries-and-cha...

I last used HAProxy about a decade ago for fairly simple load balancing and caching - worked really well.

Question about the suitability for a dynamic backend case. I have a system where clients are assigned to backends exclusively for the duration of a session. At the moment I have a pool of backends and each client is explicitly told which backend to use so they can route to it. Ideally I’d make this transparent so it was easier to reassign on the fly. Is this case reasonably supported with HAProxy and which features should I be looking at to get a sense of it?

I’m working under the assumption that I’ll need to build the machinery to maintain the pool and registry of where clients should be routed to.

While I have not tested this specific use case, I would start by exploring map files [1] and/or stick tables [2]. Map files can be updated on-the-fly through the HAProxy Runtime API. With stick tables you can persist a session/client based on anything found within the request headers, which would include a users session id.

[1] https://www.haproxy.com/blog/introduction-to-haproxy-maps/

[2] https://www.haproxy.com/blog/introduction-to-haproxy-stick-t...

Excellent, thanks for the info, will check it out.

Cost. I imagine.

To expand on this I believe an aws load balancer is $30 / mo.

For early stage projects, and particularly for personal projects it's more cost efficient to use haproxy.

As an added bonus it's got all the bells and whistles to match demands as you scale, and no (cloud) vendor lock-in

Missing capabilities. Some cloud provider load balancers don't support mutual authentication with TLS, for example.

They have very nice features that provider's load balancing might not offer you.

First to me is the ability to retain the IP address. When using AWS Load Balancer, the LB scale automatically and replace the IP.

Second is sophisiciated routing. Example route by cookie. What it buy is that by setting a cookie you can make sure the request hit a specific pool. You can also have concept of backup

Third is lots of useful metrics with their stats page.

Fourth is the ability to alter the responses such as removing header. You have to pair with lambda edge for these on AWS. What it helps is imagine you can write back the user_id to HAProxy, HAProxy log it together with the request, and delete it(so client won't see it). It helps to easily tail the log for a specific user.

And a lot of rate limiting built-in features that just take a few lines of config.

Small note - AWS Network Load Balancers maintain the same IP addresses.

Yeah I was confused too. Why on earth will someone provide a load balancer if it keeps changing it's IP address as it scales?

The normal AWS load balancers are set up to stay in service as long as the cache time on their IP address having been served by DNS... (which, FWIW, I have long argued "isn't sufficient", as many ISPs in countries half a world away from the US or using tech like satellites don't honor DNS cache times and then your requests end up getting routed to someone else... I routinely got tons of HTTP requests clearly destined for someone else's product).

Because they're intended to be referenced via hostname, and yes, regular HTTP AWS load balancers do indeed rotate their IPs.

That's true, but functionality-wise, they can do much fewer things.

With an application load balancer, you can set up multiple backends, do basic redirections, do OIDC authentication, etc.

With a network load balancer, you can only spread TCP and / or UDP connections.

To add to what other comments have already mentioned - cloud lbs send you random resets. If you don’t want to have your tcp sessions fail randomly under normal operation you have to use your own load balancer

We actually use both. We have the cloud provider's load balancers balancing to HAProxy load balancers. We use HAProxy, because it is much more flexible and powerful the the cloud provider's LB, and we use the cloud provider LB because it integrates because it is simpler to terminate public TLS with them and manage the certificates.

Looking at the standard haproxy error pages present when using an Azure App Service, you might be surprised to learn that the cloud providers are using haproxy for you, behind the scenes.

We use it at work for hosting an app designed pre-cloud (using one db per tenant) in AWS:

We deploy auto-scaling groups for each major app version (because customers can choose when to upgrade, so we can have anywhere from 1-3 versions live at a time). There's a database that has an entry for each customer system, with one or more domains, the app version and the db name (usually auto-generated but for historical reasons can be set manually). There's a UI to manage all this.

A script takes this data and builds haproxy config, creating backends for each version group and routing rules for domain to the proper group. This part could maybe be done with ALB now, but I am not certain of that.

We also automatically configure SSL for all domains: anything that doesn't have a static .pem file gets LetsEncrypt cert. Most of these are done via HTTP-01 because they're customer-owned domains that just CNANE to us. None of this is doable via AWS built-in stuff.

There's also a bunch of other hacks that haproxy does:

doesn't redirect to SSL for a couple specific (non sensitive) URLs+user agent that doesn't follow redirects;

returns a fake "success" page for a long-gone service called by an obsolete client some customers of (ex-)customers are still running, which effective causes a DDoS attack due to retries if we return 404 or 5xx;

Has some awareness of backend state and shows better error pages than just a generic 5xx depending on situation.

Haproxy instances are behind NLB, but otherwise there's a single (layer 7) hop to the app server.

The end result is you can configure a new system via our management UI, and so long as DNS is setup (using a wildcard subdomain we own, and/or customer's CNAME entry) within a few minutes it will be live (database deployed, app servers aware of connection/domain mapping, proxy configured with SSL).

We love haproxy. It is the load balancer powering millions of requests per second in aggregate in our data centers. These data centers cost less than half what the cloud would.

The .org[0] continues to be the better site.

[0] https://www.haproxy.org/

I can only imagine some investment banks have donated a fair bit given that HAProxy now has built in support for FIX. I really wasn't expecting to see that!

Since ~2004, folks in the financial industry have been migrating away from FIX to other lighter protocols such as ITCH, OUCH, and SBE (standardized from FAST: "FIX Adapted for Streaming").

This is analogous to the move we're all witnessing from HTTP to HTTP/2 (standardized from SPDY) and HTTP/3 (QUIC).

The desire to reduce the amount of bandwidth spent pushing uncompressed text around and reduce the amount of CPU needed to parse strings is what spurred market participants to explore alternatives to FIX, namely in the form of binary encoded message that costs less to parse (from a CPU and bandwidth perspective).

As libraries release support for alternative protocols and the industry compiles their projects against these newer versions, the adoption of non-FIX will keep increasing.

Similar to how you get HTTP/2 for "free" when using the Go stdlib so its downstream consumers such as Caddy and Traefik can provide that as a default to end users.

https://en.wikipedia.org/wiki/List_of_electronic_trading_pro...

EDIT: Added year for context on timeframe of first implementations

I've been long interested in HAProxy, but I can't seem to sell myself on it. When might I choose HAProxy that OpenResty[1] would not be sufficient or better?

[1]: https://openresty.org/en/

IMO you would use OpenResty as a web server and HAProxy as a load balancer in front of it. HAProxy focuses on being a high-performance load balancer. OpenResty is built ontop of NGINX and many of the load balancing features in NGINX are locked behind their "Plus" product, all of which are provided by default within HAProxy. In order to take advantage of these features in OpenResty (such as active health checking and DNS service discovery) you would need to use Lua scripts where in HAProxy they are implemented in the core.

HAProxy as a load balancer / load shedder / traffic shaper is what I've seen it used for.

Disclosure: Community contributor to HAProxy, I help maintain HAProxy's issue tracker.

HAProxy is virtually unusable as a file server, but really shines as a reverse proxy. One of the main reasons I almost always deploy HAProxy in tandem with (stock) nginx is the superior HTTP rewriting / configuration. See also my comment back on HAProxy 2.0: https://news.ycombinator.com/item?id=20198232

HAProxy's configuration is a procedural style configuration. The HTTP rules (e.g. adding or deleting headers) are processed in order if their condition matches. (Stock) nginx has a declarative configuration which makes it hard for me to understand which options apply when and the inheritance by nesting different blocks is sometimes very confusing. One example would be an `add_header` in a `location` completely overriding (instead of extending) an `add_header` in a `server`. So if I want to put e.g. HSTS into the server to apply to the full host and also want to add headers for different paths, then I need to duplicate the HSTS stuff.

Any blog post or release announcement that is expected to get wide dissemination should preferably have a blurb that explains what the project/product is, so that the post is accessible to more people. Maybe something like "The Reliable, High Performance TCP/HTTP Load Balancer".

HAProxy is so well known that it really doesn't need to. I mean, you wouldn't expect Facebook to mention the fact that it's a social network on every one of its blog posts.

And if you don't know of the product, just Google it.

It is a matter of choice. Do you mind adding a sentence for the people that are too lazy to do some search? Maybe if they are too lazy they are not interested in the product, but the cost of that sentence is negligible. Even Facebook was a lot less popular at some time and my mother asked me what is that.

The cost of the sentence is screen real estate, plus a distraction from the purpose of the post.

I sometimes feel the collective of spirit of HN can never be satisfied. If you do put that sentence in, someone will probably question why it's there :p

kzrdude, great feedback. These big version announcements gather the attention of people who may not be familiar with HAProxy and so, I'd say they are a unique case where we should introduce the product for newcomers.

HAProxy is great software, mostly because Willy can focus.

But I feel sorry for him now, with HTTP/2 and HTTP/3 the pain will never stop. And IPv6 makes most of this obsolete if it happens!

I stick with HTTP/1.1 since I'm building a native 3D MMO; only Microsoft can block port 80 which I doubt they will atleast in the coming 5 years!

Linux wont, they are not that stupid.

Edit: allready the bots have downvoted, this is beautiful.

I can't tell whether I'm missing something: how would IPv6 obsolete their work?

The only "real" reason you need a loadbalancer is that you want to save IP addresses.

If all your machines have their own unique public address you can loadbalance with your own software by pointing people directly to different machines.

> The only "real" reason you need a loadbalancer is that you want to save IP addresses.

What?! Load balancers tend to be in place for availability and scalability reasons, not to save IP addresses. Sure, there are many use cases where you might not need to have public IPs on the backends a load balancer is sending traffic to, but to say that's the only "real" reason for a load balancer seems to be missing the point.

A load balancer is, for example, used to distribute traffic to multiple worker nodes to scale horizontally.

It can also be used for canary deployments where you roll out a feature to 1% of your user base.

There are several other use cases. I'm pretty sure 'saving IP addresses' is not a use case you'd use a load balancer for.

> The only "real" reason you need a loadbalancer is that you want to save IP addresses.

NAT. You're thinking of NAT.

Even with scarce IPv4 you could currently do that just running backends on different ports

It's common to do internal services that way but you can't control how or who or the internet might try to connect to your service so the load balancer offers a layer of protection so servers can't be picked off individually.

They can also offer an abstraction/routing layer so you can do things like canaries and controlled rollouts which would be much more difficult just advertising IPs directly to the client.

I'm going to reply to all of you here:

You cannot use different ports from the client, it has to be 80 for HTTP, I meant; do you own loadbalancing from the client!

So f.ex. the browser would load parts of the site from another server via javascript or redirect to another dns with another IPv6 address.

To have a bottleneck between all your clients and all your servers is not good architecture.

> To have a bottleneck between all your clients and all your servers is not good architecture.

Maybe not, but everyone else is doing it. If your clients are web browsers, it's hard to convince them to do useful things like notice servers have changed; much easier to focus their attention on a handful of load balancers on static IPs and figure it out from there.

Reply on news.ycombinator.com