Microservice Discovery with SkyDNS | Hacker News Reader

Microservice Discovery with SkyDNS

blog.getadmiral.com

57 pointsfastest96323 comments

I'm excited for all the buzz going on around service discovery, it's often one of the most crucial design decisions for a SOA architecture. I'm really bummed, however, that so many of them rely on clients doing DNS lookups. I vastly prefer a system like airbnb/Yelp SmartStack, consul template, or really any system that reflects discovery state in real load balancers such as HAProxy, nginx, or even those new fangled service load balancers like vulcand.

Honestly just don't do DNS. Why?

1. Performance

Service discovery should not happen during the life of a HTTP request. The number of times that I've had to debug performance problem that ultimately ended up being DNS requests taking like 100ms (which for most internal applications is too damn long) is staggering to me. Yes you can mitigate this, but it is not easy because even local DNS has to miss the cache sometimes.

2. DNS propagation sucks.

This article talks about one side of the DNS propagation problem ("it's down" -> removed from DNS), but they seem to completely ignore the other side of the DNS propagation problem (removed from dns -> clients stop talking to it). At any kind of reasonable engineering scale one of your libraries (watch out for ones that cache DNS "for performance"), programming languages (java), or infra tools (nginx/haproxy) are going to not re-resolve DNS like you expect them to.

3. More Performance

Client side load balancing is legitimately hard, and if you're just doing round robin over DNS records you are likely screwing your high percentiles. HAProxy/nginx/vulcand/etc all support better load balancing strategies like least connection load balancing that really make a difference here. This can be the difference between a 500ms 99% and a 50ms 99%.

These are very good points. I'll be honest, I don't have a lot of experience with SmartStack, but I'll be reviewing it.

As far as caching, ideally you'd have no caching, however, certain optimizations can be made on this front, such as combining in-flight-and-duplicate queries.

Load balancing could be done via the SRV records, but that would be difficult since it requires that you republish the SRV record. This could be done every minute or so based on load but not on a per-request basis.

Thanks for your insightful comment!

on point 1, I've not seen local authoritative DNS take that long to resolve a local address.

If the local DNS has to recurse out to find an address, then yes. But if thats the case, then something has gone wrong with the setup. (this is local service discovery after all.)

on point 2:

Yes, long running processes cache DNS. so does SSSD and nslcd. However in the world of containers, the first point of call for debugging appears to redeploy, so this goes away, by accident. (and not entirely convincingly)

on point 3, yes, yes very much.

SkyDNS is pretty nice, but I'm not sure I see a huge value in layering DNS on top of etcd for microservices, unless you also standardize the SRV format being used. Why not just use etcd directly?

After all, the format has a proprietary format/structure that no tool (outside the SkyDNS sphere) knows about. So you've already invented a new protocol on top of DNS. You're not able to leverage any existing tool/library out there that resolves names via DNS, because they can't use your information. (SkyDNS can of course be used to look up A/CNAME records, which I totally agree is nice, but as more of a backwards-compatibility layer kind of thing.)

You do get some value from libc's name resolution, and you can institute a policy across your apps that they resolve subdomains relative to the domains listed in resolv.conf, so for one app to talk to another, they just look up "fnord", which resolves to "fnord.dc-east.production.example.com" or whatever. But I'm still not sure I see a lot of value over a client library that talks directly to etcd. You're going to want a client library anyway, to deal with those SRV records.

You bring up good points. The big advantage I see is the ability to fallback to standard tools if you don't want to use SRV. If I want to debug something I can just type "dig SRV myservice.mydomain" and see the results. I can also just use standard DNS libraries to do just regular A lookups if you don't care about priorities. For anyone else that does, we've released a bunch of libraries so they don't have to do anything to start using it.

I also wouldn't say that SRV records are proprietary, they're an official record-type (that many DNS providers support).

Personally, I just don't see why anyone should be creating their own format when some other established and compatible one exists. DNS is already widely used for domain/website discovery, why should that not be extended to inter-datacenter discovery? That's obviously rhetorical for this conversation, but that's what we asked ourselves.

Ah, you're right — for some reason I was under the impression that the format of a SRV record was specific to the protocol, but apparently it isn't. My thinking was confused by the fact that SkyDNS apparently doesn't use the standard name format, as specified by the RFC:

    _Service._Proto.Name TTL Class SRV Priority Weight Port Target

[1] https://tools.ietf.org/html/rfc2782

SkyDNS supports whatever naming scheme you want to use. You set the names yourself. That means it can be compliant with RFC2782 if you choose, or the names can be semantic -- like Kubernetes uses, with each position in the name implying a semantic meaning "servicename.kubenamespace.domain"

Disclaimer: I invented SkyDNS.

As bketelsen says, you can use any name format, but that specific format isn't strictly enforced. In Kubernetes, that is the format used when using the DNS addon (which is powered by SkyDNS).

I know this might sound odd, but ETCD and DNS look pretty similar.

They are both hierarchical distributed key:value databases (well if you ignore SRV and TXT records...) Both have a system for time limiting keys.

However etcd comes with a REST protocol built in, and higher latency.

I'm not saying that DNS is better than ETCD.

Well, adding a library for supporting SRV records is not simple... If you use Weave [1], it gives a unique IP address to every service instance and a DNS record, all with zero configuration. That means you just stick any service in whatever default port it has, and it's all good. You can also have round-robin load balancing through DNS for free.

[1]: http://weave.works/net

The performance penalties seem pretty significant at least as of a few months ago. Has that changed?

SkyDNS is a very useful tool, but the name is very confusing.

Everytime I read "SkyDNS", I think it's about skydns.ru

IMO - better to have etcd drive Nginx/HA Proxy via confd. This is flexible, fast, and performant. While it could be a coo idea to push load balancing logic entirely to the client to eliminate a load balancing tier altogether - DNS isn't really designed for it.

Definitely a cool idea! And it sort of works, but then if you need to remove failed services quickly you need to use a non-DNS interface for it :)

Failed services are automatically removed when the associated service "dies" or closes its connection with skyapi. If you're using holdingpattern then it's removed whenever the service stops. This is only because we wanted the removal to be quick and automatic.

What's wrong with actual DNS?

I'm not sure what you mean about "actual DNS" but I'm going to assume you mean A records and not SRV records. A records are available for all services as well with this set up we just preferred to use SRV so we can get port information as well as weighting. If you want to stick to just A records that's definitely an option.

I read this as just "actual DNS". An actual DNS server, that serves the records you need, like SRV records. Eg. djbdns[1]. It's not like using DNS SRV for service discovery is some crazy thing; Kerberos uses it, and by extension Windows AD[2].

Other "recent" uses of SRV is XMPP server discovery.

Anyway, SkyDNS might be great -- but is it tested/aimed at being better/faster than other battle-tested DNS servers? Everyone needs to host DNS in some way -- can they standardize on one server, and just use SkyDNS? Should they?

Or is the main thing just wrapping a REST api around DNS? [ed: like [3,4]]

Even if I need/should use a non-standard (non-authorative, not connected to the Internet root) DNS infrastructure for service discovery, do I need to turn off caching if I want to use SkyDNS for it?

A lot of these kind of questions leap to mind.

To draw a parallel, one can use nginx for TLS termination, reverse-proxy and as a host for static assets -- often it'd be good to set them up as different services, even if they run the same software. It a) allows easy switch if needed (maybe HAProxy turned out to be a better load balancer), and easy scalability. But all else being equal, without any major differences in performance (order of magnitude), or feature set -- having two instances of one piece of software is likely simpler than having one instance each of two pieces of software.

So, in that light, "What's wrong with actual DNS"? :-)

[1] http://anders.com/projects/sysadmin/djbdnsRecordBuilder/

[2] https://technet.microsoft.com/en-us/library/cc961719.aspx

[3] https://github.com/ajclark/bind-restapi

[4] https://doc.powerdns.com/md/httpapi/README/

Well an actual DNS server, Whats wrong with authoritative DNS? just use DNS NOTIFY to update slaves on record change.

That way TTL is not a problem, unless you have SSSD or nslcd doing caching on the client.

It'll remove a massive layer of complication, and will mean that you can have redundant service discovery even if your ETCD/skydns fails.

It'll certainly scale much, much higher than skydns. if you don't want to create your own DNS, Use Dyn. It has a REST interface, a very nice SLA. if you query the servers directly you also don't have to worry about TTLs

Right, if you know how to set it up or have infrastructure in place... But most developers don't have time to read 600+ pages on how to run bind server OR simply have ops team that is very conservative at what goes in the DNS land.

just buy it in? dyn have a REST API.

failing that, bind is not difficult to learn. If you have an artificial divide between your devs and you, then you have far bigger issue. If you can't convince them of the merit of using DNS then there really is no hope.

the whole point of DNS is that you can delegate subdomains, so you can neatly isolate zones from each other

Plus saying something looks hard is a terrible justification for not trying something. I know bind isn't trendy, but it works and is simple. Failing that, there are at least 3 companies out there with REST APIs and 100% uptime SLAs.

prototype all the things!

Can the SRV record be pushed via DNS Notify as well ?

This is the main issue with DNS. That you need to get the port information from somewhere.

DNS NOTIFY is a mechanism to update slave servers.

if you switch allow-notify on in bind it'll push updates from the master server to n slaves. This updates all records, TXT, AAAA, A, SRV, etc

Reply on news.ycombinator.com