HAproxy in the era of Microservices | Hacker News Reader

HAproxy in the era of Microservices

47ron.in

182 pointsthorinus62 comments

My current project consists of ~7 microservices, which has been wonderful for many reasons but exposed us to substantial complexity on this score. At present ~2 of them have to receive communications internally or externally over HTTP, so we have those fronted by a proxy to make service discovery easier. (It is, regrettably, not an off-the-shelf proxy, since one of them required application-specific routing to a sticky host(s) depending on the contents of the HTTP request).

The thing which has made our lives MUCH easier from an orchestration, discovery, and routing perspective is NSQ, an OSS message bus. It's an event model rather than a request model -- producers of events say "Hey world, by the way, EventType just happened." and consumers can register to say "Apropos of nothing, please tell me about any EventType" without producers or consumers having to be aware of each other or aware of what a consumer intends to actually do with regards to the event.

Events go into topics ("a type of event") and each topic can have an arbitrary number of channels ("one thing which should happen for each event in this topic"). If you're clever about where you raise events, you can stitch together arbitrarily complex systems by just adding new consumers on new channels, without having to make the producers aware of the new functionality at all.

Concrete example: We wanted a log of all orders made to the stock exchange. We have a little utility, nsq-to-s3, which takes all the events on a topic/channel, persists them in memory/disk for a while, and periodically flushes that log to S3 for long-term storage. This trivialized the logging feature. (I was going to write our own nsq-to-s3 but an OSS project of that same name existed and was adequate to our needs, so we didn't have to go to 8 microservices... yet.)

This was much easier than making a generic S3 logging service and then making edits to seven microservice codebases to make a HTTP request out to the logging service, with e.g. retry/failure recovery/monitoring/etc, everywhere in those codebases we wanted something logged to S3.

This sounds similar to kafka, how does it compare?

I have heard from several folks that they use Kafka where we'd use NSQ, but have insufficient experience with Kafka to give an informed opinion on the tradeoffs. I will say that few technologies I've used in my career were as easy to adopt as NSQ was.

Kafka makes a central promise that NSQ doesn't, that of delivery order. This is a complex promise to maintain in a distributed system, and this complexity is apparent in operating a Kafka based system.

This is how Hailo https://sudo.hailoapp.com/services/2015/03/09/journey-into-a... , Mondo https://getmondo.co.uk/ and a few others have approached the orchestration. Hailo uses Rabbit, Mondo uses NSQ.

It really starts to shine when you add controls to the consumers to enable soak testing of new services by either redirecting some small % of traffic to the new version dynamically.

> It is, regrettably, not an off-the-shelf proxy, since one of them required application-specific routing to a sticky host(s) depending on the contents of the HTTP request.

Did haproxy not support the necessary ACLs for you to accomplish this? It might be possible in 1.6, which lets you process the HTTP request body in addition to the headers.

http://blog.haproxy.com/2015/10/14/whats-new-in-haproxy-1-6/

Do you have a centralized nsqd instance, or do you have a 1:1 relationship between nsqd instances and producers?

One nsqlookupd, one nsqd per machine which hosts 1+ producers. We'll eventually have a small number of nsqlookupd.

Tangentially, but is there a queueing service with good and fast retrying of messages with no destination?

Basically I'd like the incoming router to send a message of a specific type and subtype to an existing listener, but if one does not exist I'd like to take back the message, spawn a new listener which will get on the queue and resend the message.

I wonder if I could use a messaging service for keeping track of this, rather than have to manage my own map of what application backends are ready to process which type of messages ( spawning that backend takes a lot of resources so I want them to be born and die based on demand).

Curious to know how you handle dependencies between services. For example, say you have a User service, I could imagine creating a client library for interacting with this service (the library being responsible for knowing where the http end point is, serializing/unserializing JSON into domain objects, etc.). Then any service wanting to interact with the User service just adds this library as a dependency.

Along those same lines how do you deal with fetching of domain objects and associated relationships from these services? For example, let's say in one instance you want a user and their manager and in another instance you want a user and their manager's manager and so forth for any relationship that a user might have (that the User service is responsible for).

The most significant issue with HAProxy and large SOAs is the lack of really good dynamic re-configuration. You don't want to have to manually change your configuration every time someone wants to add or change load balancing to a service, and if you do it automatically you run into HAProxy reloads being a reliability problem as they drop traffic on the floor due to Linux being goofy in how it implements SYN handling.

It looks like there is a lot of good advice on this thread for how to mitigate the configuration problem (bamboo, consul template, etc ...) but then you run into reloads, and even if you use zero downtime restarts as I described in my HAProxy deep dive (http://engineeringblog.yelp.com/2015/04/true-zero-downtime-h...), you still end up with periodic ~50ms blips in your timings.

We've found that Synapse (https://github.com/airbnb/synapse) is best in class at managing HAProxy for us in a fairly large SOA. It is really intelligent about doing as much as possible dynamically over the HAProxy stats socket as opposed to reloading for every configuration change, and we can finely tune how often we are reloading HAProxy. The best part is that it is fully pluggable. Use DNS, marathon, zookeeper, etcd, etc...: Synapse can manifest that service registration information into HAProxy configurations.

I personally prefer running a large number of logically separate HAProxy instances (a pair for each service). This makes it easier to forecast / tune request volume of the proxy servers on a per-service level, and it limits the impact that reconfiguration of one service can have on another (which helps immensely in root-cause analysis of production incidents once you've scaled beyond one team). It makes firewall rules a pain, but really no more painful than the general orchestration / configuration pain you feel at this scale.

The good part about HAProxy is that it's flexible and can be configured any number of ways, but I've found that flexibility can weigh some teams by using it to solve problems that it's really not designed for. Sometimes it's better to use a brute-force approach with a simple configuration and deal with the complexity in a tool better designed for complexity like Mesos.

On a side note, Kong (https://github.com/Mashape/kong), also mentioned in the article, allows re-configurations through its RESTful interface for every plugin.

Another problem with the configuration reloading is that all the backend/service metrics counters that HAProxy exposes on its metrics endpoint get reset on every reload. That makes for really noisy metrics gathering in situations where you have rapid successive changes to the config, such as gradually scaling up hundreds of instances of a new version of a microservice and then scaling down the old version. When you then scrape the metrics via Prometheus using an bridge like HAProxy Exporter (https://github.com/prometheus/haproxy_exporter), you lose a lot of counter increments due to the frequent resets, so you get lower and noisy rates for that time period.

I hadn't heard of Prometheus before. We use nagios for alerting/alarms, and graphite/grafana separately for metrics. Would you recommend looking into prometheus as an alternative?

As a Prometheus author, I'm biased, but yes :)

See also http://prometheus.io/ and https://www.youtube.com/watch?v=HjN23GgCzQY for an intro.

HAProxy is a great fit for most microservices, but I'm not sure it has the feature set for most organizations that are deep into microservices - at least in the way the author describes it. The problem is more business-driven than technical; so let me explain why (and feel free to correct me if you think I'm wrong!)

Most companies don't need microservices, so I'm making an assumption that any company using microservices has both the volume and complexity of business partners that make a microservices approach a good business idea (this assumption is assuredly false, but still useful). And if you've gone this route, the next step is generally organizing your business around an SOA where business units can be represented by an API (microservices are useful from a business perspective because you have many stakeholders within a company that want to make small changes frequently but independently). Once you've done that, you start wanting to encapsulate your relationship with your customers in an API as well, and the eventual destination (if your company survives long enough and is successful) is that you just operate a set of APIs that power your customer-facing UX product in addition to the API-based services you provide to your high-volume clients.

My point is this: the end state of a microservices architecture is an API gateway. API gateways provide essential services such as authentication, access control, and connection throttling. You still need something like HAProxy to provide you with load balancing, health checks, etc. across many nodes - but you're going to have a very simple instance of HAproxy for each microservice and rely on your API gateway to organize everything under a single domain.

With microservices, your configurations can get incredibly complex very quickly, so it's often better to maintain independent components with very simple configurations (that can be easily updated dynamically) than to try to have a single component in the middle of your stack upon which many other components are dependent.

I think that HAProxy can work, but it needs assistance. As it doesn't allow you to use resolvable hostnames, you can't rely on DNS load balancing for microservices. As I said in my comment on the article, Consul + Consul-template works well for this.

Yeah, you need an orchestration layer to really handle microservices at scale (yet another reason they're a pain in the ass!) Not just for HAproxy, but there are countless other situations where you need to use referential data to build configurations on the fly -- which is what an orchestration layer does.

As of 1.6 it does allow resolvable hostnames [1]. This is the example given in the release announcement:

    resolvers docker
     nameserver dnsmasq 127.0.0.1:53
 
    defaults
     mode http
     log global
     option httplog
 
    frontend f_myapp
     bind :80
     default_backend b_myapp
 
    backend b_myapp
    server s1 nginx1:80 check resolvers docker resolve-prefer ipv4

[1] http://blog.haproxy.com/2015/10/14/whats-new-in-haproxy-1-6/

Oh, cool, that's interesting!

Thanks for the heads up. Missed that one!

That's pretty new; but still doesn't remove the need for an orchestration layer (though it does make it a lot easier since I often rely on DNS with short TTLs for global routing).

I'm sorry to be that guy, but the lack of contrast makes the text difficult to read.

It's good enough

I'm sorry to be that guy, but the lack of contrast makes this comment difficult to read.

Try one of these bookmarklets to make the page look better - I routinely use zap or zap stylesheets to get rid of all annoying page elements and stylings:

https://www.squarefree.com/bookmarklets/zap.html

Yes, it's annoying. However you can use bookmarklets to fix it and you can also use "reader" feature if you're on Firefox, it works surprisingly well (on most pages - not on HN unfortunately).

Folks should take a look at Baker Street (bakerstreet.io) which is a HAProxy-based system for routing traffic between microservices. Baker Street is designed to be really easy to get up and running (the install is super quick) and is based on the Yelp popularized SmartStack. However, one of the beautiful aspects of Baker Street is that unlike SmartStack it does not require running and maintaining a ZooKeeper cluster which can be a PITA for teams just starting to work with microservices.

Full disclosure: I am the lead developer of Baker Street :)

Website: http://bakerstreet.io

Source: https://github.com/datawire/bakerstreet

Hey there!

Just a small nitpick: Running Zookeeper is insanely simple. We deployed it in 2-3 days, and it has had literally 0 issues over the last year.

I highly recommend https://github.com/mbabineau/docker-zk-exhibitor to get started (Docker container that wraps together Zookeepher and Netflix's Exhibitor manager for it).

Remember, pick boring technologies that just work.

Thanks for the response! I agree that ZooKeeper is not exceptionally difficult to run, but it does bring in some additional overhead and complexity for a development team that is just trying to get started with microservices, for example sorting out the deployment and quorum bootstrapping. For a team that might not have an experienced systems or infrastructure engineer it could become a nightmare. By bringing in Exhibitor you've added even more initial complexity to the deployment just so you can get started gluing services together. That's what we're primarily trying to solve with Baker Street at this time: super simple deployment and usage; minutes versus days.

zookeeper is not a boring technology that just works:

http://blog.cloudera.com/blog/2014/03/zookeeper-resilience-a...

http://arstechnica.com/information-technology/2015/05/the-di...

https://tech.knewton.com/blog/2014/12/eureka-shouldnt-use-zo...

Here's ibm and netflix with some advice about how non-boring running a ZK cluster can be:

https://www-01.ibm.com/support/knowledgecenter/SSCRJU_4.0.0/...

http://techblog.netflix.com/2011/11/introducing-curator-netf...

Looks neat! Any plans on adding support for Consul as a directory/backend? (understand if you'd hold the same arguments against it as for ZooKeeper)

Thanks!

We have had very abstract conversations about adding alternative back ends, but presently we are not working on support for that feature because as you correctly noted, at least for Consul and ZooKeeper it is somewhat against the operating philosophy.

We're not opposed to it though and if you open a feature request we'll track interest which will help us decide direction. Baker Street is not a secondary component for some other system in our company. We consider Baker Street a core product even at this early stage in its development.

This, from your page, is worth highlighting:

> because we wanted a simple, highly available service, not a strongly consistent one.

I absolutely agree with this design choice.

This is the big flaw for me with how a lot of setups use Etcd, Consul, Zookeeper and the like.

A lot of the time they're being used because we need high-availability, not at all for their consistency guarantees, and their consistency guarantees has great potential to reduce the availability.

I've not run Zookeeper in production, but both with Etcd and Consul I've had situations where the clusters have gotten into totally broken states because the cluster membership has changed in bad ways. It makes sense for the guarantees they are providing. It does not make sense if you don't need the consistency guarantees.

The way I see it, lost service registrations are a non-issue as you generally want to continuously renew them combined with a TTL to avoid stale registrations, so the occasionally lost one due to e.g. a part of the service directory cluster failing at a bad time is no big deal. Conversely, if a service stays in the service directory when it shouldn't, the worst that happens is that you run into an unhealthy service when you try to issue requests, but that will happen during normal operation while using services with consistency guarantees anyway, since the service may have failed between health checks anyway, which means you still need to handle retries.

I'd rather use a service-discovery solution with fairly significant consistency issues that keeps replying "no matter what" than one which focuses on consistency but may stop answering.

We use a single monolithic HAProxy to route to all of our applications. We have about 8 different kinds of applications we host. Of those, we have a few separate instances of those applications available at similar (but different) URLS. Think: `https://sub.domain.com/<prefix>/appname/path`.

We route based on `sub`, `prefix`, `appname`, and even sometimes `path`. Managing the config is literally going into this relatively big file, adding in ACLs, adding backends, and setting up the routing rules. We manage the file with puppet, but it's static - not generated.

I feel that it's probably time to start deploying many HAProxy instances, generating the file per service, and then maintaining a frontend haproxy that routes according to subdomain.

Can anyone comment on similar deployments? Have you found it easier to manage many haproxy instances rather than a single monolithic one?

We're using a home baked solution using our own config server (https://github.com/niko/goconfd) which evaluates templates POSTed to it. Combined with some trivial shell loops over a blocking POST and a HAProxy restart this does the job sufficiently well. If anybody is interrested in the details I'm happy to elaborate.

I heard AirBNB installs HAProxy on each of their nodes, and then manages the configuration for each service on each node. Then the service just needs to access 127.0.0.1 instead. This sounds compelling to me because you eliminate the single point of failure of the HAProxy goes down.

It's called Smartstack and is also entirely open-source - http://nerds.airbnb.com/smartstack-service-discovery-cloud/

Having been a long-time user, I can tell you the added complexity of every service having to allocate its own port, then tracking that port across every other service is a lot of overhead on its own.

Not to mention you also have to run a zookeeper cluster and two additional daemons on every instance.

All in all it's a good concept, but slightly too complex for my needs. Recently made the switch back to using internal ELB's for every service and not looking back.

I'm moving away from HAProxy. The lack of zero downtime config reloads is consistently causing problems, especially in a high reload environment (i.e. using bamboo [0]). One team deploys an application that goes into a restart loop, changing our routing table constantly? Well there goes our 99th percentile out the window.

That said, would prefer not to be using nginx, so if anyone has any recommendations for high quality zero-downtime-reloadable HTTP proxies, I'd be very interested.

[0]: https://github.com/QubitProducts/bamboo

Check out the deep dive I wrote into how you can help prevent HAProxy reloads from dropping traffic without a large 99% hit over at http://engineeringblog.yelp.com/2015/04/true-zero-downtime-h...

Looks like bamboo encourages folks to use this strategy via a custom reload command: https://github.com/QubitProducts/bamboo/issues/152

That's egress traffic :) And yeah, https://github.com/QubitProducts/bamboo/issues/143#issuecomm... is currently my best idea (with some additional nfq stuff). Doesn't help when requests span the entirety of the reload though.

I mean, I wouldn't encourage anyone to use HAProxy for microservice load balancing unless they're running it on every node, at which point clients are always talking to localhost and everything is egress.

That being said, you can turn ingress into egress with an ifb. A coworker of mine created a proof of concept for using our strategy with an external facing load balancer but I don't think he ever tried it in production so I'm not sure how well it works.

To be fair I'm scared of both the iptables and tc solution on external LBs because you might never know that you're refusing connections accidentally. Linux 4.4 is coming with some patches that should help with this and afaik BSDs have done it right since the start.

AFAIK Apache Traffic Server does zero-downtime reload http://trafficserver.readthedocs.org/en/latest/reference/con...

I'm curious whether https://github.com/eBay/fabio would meet your needs. It picks up the routing table from consul changes and service registrations and reloads it without restart. All the services have to do is to register their routes in consul. It is meant as a replacement for consul-template+HAproxy.

Disclosure: I'm the author

Unfortunately consul integration isn't useful for us.

Live configuration via the stats socket is frequently overlooked, but works a treat. I'd recommend investigating it before giving up on haproxy.

I use it post reload to persist the state of health checks, which I thought was pretty cool. However, I don't believe you can add a (new) server to a backend via the stats socket.

> However, I don't believe you can add a (new) server to a backend via the stats socket.

Yes, though a bit of advanced planning and creative configuration can usually work around this limitation.

Making the assumption that new servers and services occur within a predictable range of IPs and ports (or have their own haproxy to route within the server), you can create a list of backend servers which are all disabled and just waiting to be individually enabled.

I've done this in the past, and the disabled server entries never seem to have a negative impact on HAProxy. It also has the advantage that you can set your HAProxy instances to peer with each other, so the change only has to be made once to propagate to connected instances.

This is exactly why we built out conf-builder (https://github.com/radiantiq/conf-builder). None of the existing solutions matched what we needed and so far it's worked out pretty well for us. If you use docker there is also a neat tool called registrator that we've played around with https://github.com/gliderlabs/registrator

Using consul (https://www.consul.io/) in conjunction with HAProxy is another great way for service discovery and routing.

Yep, Consul-template + HAProxy is great for a dynamic HAProxy solution.

Has anyone actually done this at scale and seen how it works? We've talked about it a few times because Consul looks really cool but it doesn't appear to have the full capabilities of SmartStack yet, in particular Synapse (https://github.com/airbnb/synapse) for intelligently doing stats socket updates, intelligent reloads, or controlling how HAProxy reloads over time.

Also consul only has two levels of services right, "local" and "remote"? I'm not sure that is sufficiently powerful to describe SOA's which need to be able to differentiate services on the machine, rack, colo, datacenter, region locality? I suppose you could do what we do in SmartStack and just register the same service instance at a few different places since I think that consul can register arbitrary services.

Maybe all we need is a consul watcher in Synapse, and a consul-template for the Synapse configs, and we can get the best of all worlds. With Linux 4.4 SYN handling in Linux might even be fixed by Eric Dumazet being super hardcore (e.g. https://lwn.net/Articles/659199/) and we won't even have to worry about reloading HAProxy.

Services in consul are datacenter aware and there's a tagging concept that you could utilize for something rack or region.

https://github.com/eBay/fabio might also be an option for you.

Disclosure: I'm the author.

Looks really interesting. Will try it out!

That looks interesting. I'll check it out. Thanks!

As long as you don't have too many microservices... the polling and templating doesn't scale too well.

This is related to https://github.com/hashicorp/consul-template/issues/165#issu... although that is even more extreme.

Well it depends on how you implement it. I wouldn't imagine that each microservice has it's own haproxy-consul-template router. If it did, then sure, you're going to swamp it.

Maybe I'm not understanding the situation correctly, and granted, I've not ran it on a cluster of more than 5 machines with more than 10 microservices as we decided to use Rancher instead.

If you're using marathon, check out bamboo, it can generate haproxy configuration files from templates which can be triggered by changes in marathon. Cool stuff!

What a non-article! tl;dr you should use run HAproxy on a known host that forwards to your services.

Why not nginx?

Reply on news.ycombinator.com