DNS for Service Discovery in HAProxy | Hacker News Reader

DNS for Service Discovery in HAProxy

haproxy.com

87 pointsphil2150 comments

I've been exploring DNS PTR/SRV records for seamless service discovery regardless of whether you have access to an authoritative DNS server configured for a given local domain or not (so things simply devolve to whether or not use use multicast or unicast to resolve SRV records). Basically DNS-SD. The idea of using SRV records in general seems foreign even in the ops realm. It's awesome to see HAProxy beginning to familiarize people with the concept of SRV records.

> The idea of using SRV records in general seems foreign even in the ops realm.

I'm not suggesting you use it but (as for how it works in general) look into how Microsoft's Active Directory works. They've had the "service discovery" problem figured out for going on two decades, including preferences for "closer" servers, weighting, and so on.

I'm no MS fan by any means but AD is one MS product that really is awesome.

And how does MS do it? SRV records, that’s right.

Yeah, exactly. That's why I suggested he look into how it works (since "the idea ... seems foreign" to him).

I didn't mean to suggest that they had some revolutionary new unique way of handling service discovery or even that they were the first. I think maybe you misinterpreted or misunderstood my intentions.

It’s all good, I got what you meant :)

LDAP and Kerberos of all implementation are probably the most common use case for SRV records, most people probably don’t even know it. Real shame, in many ways I wish A/AAAA records weren’t even a thing for the vast majority of services (HTTP, I’m looking at you!)

I would love to see HTTP adopt SRV as well. Sadly the browser vendors think they're important enough that they can squat the address record all for themselves.

Last argument I heard in chrome bugtracker it was - doing additional dns lookup slowdown user experience, which is why they abandoned idea adding it

https://bugs.chromium.org/p/chromium/issues/detail?id=22423

That is what we call in Australia a "furphy". SRV records do not intrinsically require an additional lookup or round trip. Poor zone configurations may fare less well. But that is true of all poor DNS configuration. For sites that care about resolution speed, they already optimise as necessary, and they would for SRV as well.

Browsers do not own the A record and should never have been using it in the first place, so to continue squatting them and invent specious reasons for doing so is simple arrogance and assertion of unwarranted privilege.

Instead we have the multiple warts and misbehaviours of misusing the DNS for URL endpoint resolution.

Consul supports this out of the box.

And HAProxy supports Consul out of the box thanks to this new feature :)

Looks like both HAProxy and Nginx are going down the "pay for more advanced features" route:

https://www.haproxy.com/products/haproxy-enterprise-edition/

https://www.nginx.com/products/nginx/

What ? Are you kidding ? We're extremely open, we currently have 7 devs working full time for most of the stuff you'll have in 1.8, we provide all the sources with our packages (and some of our customers do their own builds), and all the stuff we add in the enterprise version is in fact a fine selection of stable enough features from the dev branch for people who need them right now without waiting for the next version to be released. When a customer asks for some specific feature we explain them that it first has to be in mainline so we can backport it and it gets a wider exposure to spot bugs. How can you be that insulting ? If it's a matter of unclear explanation on the web site, it's possible and then please report this to contact@ or something like this (I don't know the address). If it's just because you want to criticize without knowing, well, in this case I hope I'll never count you among our users!

It was an observation, not an insult. Many of us, myself included, are more than happy to pay for the software we use.

However, I would strongly consider hiring a mature developer relations manager. You're a fantastic developer, Willy, but you risk losing business by lashing out like this.

OK fine. But my business is not to sell products, it's to maintain haproxy the best load balancer. It's other people's task to present it as an eye candy thing that attracts and pleases customers. And obviously I give my opinion from both a user's and a developer's point of view.

We work hand-in-hand : if I fail at my task, the product dies and the company suffers as well. If the company fails, I will have to find another way to eat my daily steak and that will necessarily affect the project. That's why the company helps me in this difficult task by hiring the developers I need to help me on this job and why in turn I freely express my opinion when people attack the company's motivations in a way that I find really unfair.

And if you want to get a better sense of what I'm saying, just issues this in the master branch to see who fixes most of the bugs, resulting in haproxy being really usable in production by all of us :

$ git shortlog --grep BUG v1.7.0..

Nitpick: We're extremely open, ... compared to other COMPANY BACKED PROJECTS, but far away of totally OPENSOURCE projects that doesn't reserve any feature... we currently have...

https://www.haproxy.com/products/haproxy-community-vs-enterp...

Said that. Yes I'm grateful to be able to run and modify both HAproxy and Nginx for free, but parent is totally right in the comment. Both projects are going that route, compared with their beginnings.

Downvoting me, won't make those Enterprise features opensource, neither will make the parent comment wrong. Both projects are going that route.

Easy there, tiger. Not doing well for the brand of your team.

Correcting an assumption can hardly be termed as 'lashing out', maybe a 'robust' response.

Rather have founders like wtarreau speak their mind than adopt some fake passive aggressive tone or have a 'dev relations manager' spinning, at least here.

> Correcting an assumption can hardly be termed as 'lashing out', maybe a 'robust' response.

'correcting an assumption' seems orthogonal to 'lashing out'.

also, it's not clear he even addressed the op's point, insofar as i understand. haproxy enterprise does seem to contain features unavailable in the 'community' version:

http://www.haproxy.com/products/haproxy-community-vs-enterpr...

Sure but these are extra packages which help ease of deployment, management in an enterprise context etc. Some of them are side projects (eg: VRRP is provided by keepalived, route injection relies on Bird and some patches we tried to upstream and which were rejected as useless, but which we still provide in the source package). There are some internal development projects for some extra products as well (which appear in blue on the web page), and each time one of them requires any improvement from haproxy, the changes are mainlined before. Any single line of code of haproxy comes from the mainline sources, the source packages contains the patches, and all commit messages even contain the whole cherry-picked chain leading to the master tree's commit IDs first. We can hardly do better :-(

In fact, it's important to understand what enterprise users need : they never want to have to deal with patches or feature backports, most of them don't want to have to build anything, and instead they want to install regular pre-compiled packages from a permanently updated repository. That's exactly what we provide them. But providing pre-compiled packages is no excuse for not giving their sources at the same time, which we do.

Let me be very clear about something : a load balancer (and haproxy is no exception to this) is a very complex component and will always have bugs. And it happens that where it's deployed it's a critical component that must never fail, which is a bit incompatible with the first rule. For having dealt with proprietary LBs in the past, I'm well placed to know that you cannot and must never trust a non-fully opensource load balancer. And we've kept this spirit of full openness to achieve the highest possible reliability, and the GPL maintains this guarantee here. Some of the top contributors of the project are/have been working for enterprise customers and have brought very detailed analysis about certain issues, sometimes even proposing patches to fix them (or also small changes to make their life easier). As the project manager, I don't care where a patch comes from: any user, an occasional code reviewer, a customer, one of my coworkers etc. A fix is a fix, and if it addresses a real bug it must be merged, and the same rule as in the kernel applies here with no exception : mainline first. This guarantees that no patch is lost and the fact that the patch is the most possibly correct. And in order to get these fixes, you have to be the most open.

The situation as it is now benefits the two categories of users : - enterprise users don't have to wait for the next release to get certain features they need right now : some of the features from haproxy-1.8-dev are already present in HAPEE 1.6 or 1.7. So basically these users are encouraged to pick the most stable version which satisfies all their needs. - mainline code benefits from some feedback from the field (improvements and fixes) before the final release, which has allowed us quite a few time to adjust certain confusing things (option names, default behaviour, etc) before declaring them "stable" and having to maintain them forever.

The only advice I could give to people considering adopting a load balancer from a vendor : during the evaluation, ASK FOR THE SOURCES! If you can't have them, never use this product, one day or another you will regret it. And I'm not saying this just to try to place us in front when many other solutions are closed, but simply because I've been in this situation of using a closed product quite a few times in the past and hated it. That's even what led me to create haproxy in the first place!

What assumption was being corrected here? Please cite with specificity.

To me, it seems that one could read an (unsaid) implication in your original comment that is “and those advanced features will only ever be behind a paywall” — as I think in nginx’s case that’s actually true for some features (but I might be remembering incorrectly!), and by listing and comparing HAproxy and nginx in that manner I can see that implication. But that’s being kinda nit picky, though if I was an HAproxy dev and all features were developed in the open and released into the mainline for free in a new stable version in the future, I would probably read it that way and be a little miffed, I suppose

All the features developed in HAProxy, are pushed in the -dev branch of the community version. They are available for free at your own risk (running a -dev code in prod), or you have to wait until -dev become stable. HAProxy Enterprise users might already benefit from this feature in a stable way.

On the opposite, in nginx, some features are developed only in their proprietary solution. Sometime, nginx inc passes for heroes because they open source some of them...

there are separate 'community' and 'enterprise' versions, and there's even a comparison chart on the website.

You're not... he has a point.

Note, this feature will be in the free version when 1.8 is out. Early access is for enterprise customers.

no, this feature is already available in -dev community branch. So it's already available for free.

Note this feature has not yet been backported into HAProxy Enterprise...

is there official confirmation of this? it would seem like a change of pace given how the community vs enterprise feature lists play out:

http://www.haproxy.com/products/haproxy-community-vs-enterpr...

Well, nignx inc does first develop their proprietary software... Then, they "open" some features. With nginx plus, you're locked to nginx, like when you were using a F5 load-balancer!

HAProxy Technologies pushes first its developments in the -dev community branch. Check the commits for the feature given above (SRV records, non exhaustive list of patches): http://git.haproxy.org/?p=haproxy.git&a=search&h=HEAD&st=com...

HAProxy technologies simply makes the effort of maintaining a branch of HAProxy in which some -dev features are backported. That way, users of HAProxy Enterprise have the most stable and feature reach version of HAProxy. And as Willy stated, every user of HAProxy Enterprise also have access to the source packages.

So from a business point of view, the main difference between HAProxy and nginx is that HAProxy is the respects both its community and customers.

Hi there. This is really awesome news and i am delighted to hear about this.

Any change to get some directions how to test this functionality?

So far I have currently 1.8-dev2 up and running but now I find no real references about how to try it .. and if available in this repo. (maybe i have to dig into the code ?)

Thanks and all the best.

http://git.haproxy.org/?p=haproxy.git;a=commitdiff;h=e2d03d2...

Thanks and sorry for the noise

Interesting that HAProxy believes it is incumbent enough to charge for directly copying a feature Consul has had for years. When I added Consul to my stack 3 years ago HAProxy was already looking behind the times... I don't think they can catch up for any new installation.

I don't understand what you mean here. Consul is not a load balancer. It's functionality is almost entirely orthogonal to Haproxy.

If anything, a lot of people pair the two, but have so far often had to resort to cumbersome solutions for rewriting the Haproxy config to keep the set of backends in Haproxy updated.

If anything, this will make Haproxy and Consul work much better together, given that Consul serves up SRV records for the services you register with it.

To be honest it's not a matter of duplicating, catching up or anything. It's just that when a significant number of our users ask for exactly the same stuff with similar and sometimes different justifications, it becomes quite hard to avoid adding it. And if they want it there, they very likely have their own very valid reasons. Baptiste (the DNS subsystem maintainer) has a significant amount of use cases in mind and could likely much better explain than me.

We first developped DNS in HAProxy for our AWS users who wanted HAProxy to follow-up a node when it is restarted (and it's IP is changed)... That was the very first request from both community and customers. After we released this feature, the community came back with some requirements regarding the ability to use DNS resolution to resolve all the servers (or a set of servers at list) of a backend. So we had to improve a lot the first dev we did to make this possible. Support for SRV record is just the last stone we put on top of many other devs :)

We're quite proud of the result because it makes HAProxy able to scale up / down at run time without being reloaded and compatible with any service registry able to export a list of nodes delivering a same service through DNS.

HAProxy can use multiple name for the resolution, pointing to different set of DNS servers, enforcing custom "hold" timers (to bypass server's TTL or negative TTLs in case of NX, etc...) and mix all of this with "old style hardcoded" servers in the backend...

HAProxy is flexible :)

The advantage of using SRV records with consul, is that any load-balancer supporting SRV can replace any loadbalancer already in place, almost seamlessly :) This applies to any Service Registry (I tested the feature with both Kubernetes and Consul)

So yes, this feature was missing in HAProxy and we catched up because our community and some customers asked for it. So they can now use the power of HAProxy with their current consul deployments.

This was already addressed in another comment, they aren't charging for it. The preview is only available to enterprise but it will be in the 1.8 release later this year.

I think it'll become hard to explain simple things in this place. THE CODE IS FIRST DEVELOPPED INTO HAPROXY DEVELOPMENT BRANCH AND ONLY THEN BACKPORTED INTO HAPEE ONCE WE CONSIDER IT STABLE ENOUGH. ANY SINGLE HAPEE COMMIT HAS AN UPSTREAM COMMIT ID. What is difficult to understand in this statement ?

I hope the non-blocking DNS Client will be released separately as well. This seems very useful on its own

Have you tried c-ares? It's venerable and useful.

For the lazy (like me!): https://c-ares.haxx.se

Unfortunately, this is not possible! The DNS in HAProxy relies on the internal task scheduler which is itself event-driven.

Why not use GetDNS¹?

1. https://getdnsapi.net/

Anyone here ever used SRV records for local development? I really like the idea of running a local DNS server with port information built in instead of a full-blown service discovery/routing/service mesh setup.

Yes. Current setup runs a local DNS server which serves SRV records containing the local IP addresses and ports of services running inside FreeBSD jails. For instance, the payments service in one jail can communicate with the risk service running in another jail on the same host by sending an HTTP message to http://risk.local

Currently using dnsmasq but would like to use erl-dns once I finish wrapping it with an HTTP admin API.

https://github.com/aetrion/erl-dns

Yes, try Consul. It's extremely easy to configure and run.

I do agree on this statement!!! I personally validated HAProxy SRV records with both Kubernetes and Consul and I was positively supersized of how simple is consul. (Well, it does not cover all the stuff kubernetes does, you may need nomad for this purpose).

Maybe there will there be a Kubernetes a Ingress Controller?? :)

Kubernetes Ingress Controllers for HAProxy already exist. A nice one is at https://github.com/jcmoraisjr/haproxy-ingress Other examples of usage are spread around in the examples section of the ingress repo, like here https://github.com/kubernetes/ingress/tree/master/examples/d...

The DNS-SD code will ship with HAProxy 1.8, and we will gladly help out by adding it to the above ingress controller after the release. A WIP of that code is up on github in case you’re interested. In fact, HAProxy is full of surprises and little known features (like the dynamic scaling via haproxy runtime api we already contributed to the controller) and we're happy to share them with everyone!

Reply on news.ycombinator.com