Hacker News Reader

Comment by siliconc0w | Hacker News Reader

Use something like a mac address and map them logically (i.e use chef envs/roles/tags). This forces you to stop remembering hostnames and setup automation to ssh/run commands. If you try to encode information into hostnames you head into the deep end pretty quickly and it's probably going to change in six months anyway.

View on HN

I believe that's why the information is all encoded with CNAMES. That way you can change purposes easily without ever touching the original A record, which refers to the hardware (not its function)

You should be very careful with this. Even a relatively low TTL of 300 (5 minutes) can be too long to wait during a major outage. Make sure changing CNAMEs is a very rare thing, and be prepared to tolerate inconsistency.

It's also worth noting that some widely-used pieces of software cache DNS lookups in-process (MongoDB), so changing a DNS name is no guarantee that all connected processes will automatically fail-over to the new machine unless restarted.

Moral of the story: distributed systems is hard

I have run into this at least twice this week alone with both Nginx and HAProxy pointing at Amazon Elastic Load Balancers. Amazon occasionally rotates out IPs for ELBs, so everything's working fine for weeks and then boom, now we're having a bad day.

As a rule, NEVER EVER use CNAME's within AWS on Route53. You should always use their A Alias function. This provides near instant changes when you need to adjust a record. This only works on aws resources, but it is a great feature they built in.

I totally agree. In these two specific cases, R53 + ELB wasn't going to work for us though, because of reasons. (I don't think I can go into specifics, but they're actual reasons.) Our workaround was reloading the configs (which triggers a DNS cache flush on both Nginx[1] and HAproxy[2]) once every couple of minutes.

I know, I know, I hate it too. We're working on it. Just wanted to share the workaround.

[1] http://wiki.nginx.org/CommandLine#Loading_a_New_Configuratio...

[2] http://www.mgoff.in/2010/04/18/haproxy-reloading-your-config...

The JVM is another one of those things - DNS lookups are, by default, cached for the life of the VM instance.

God, how awful. That's the point of TTL!

Beyond that, my system has a perfectly functional resolver (and a daemon to manage name service lookups, using policies that our organization has chosen and assumes are being used).

I understand Java doing 'its own thing', because the goal is to provide consistent behaviour on all platforms, but it shouldn't be stupid behaviour.

I remember when Java added this feature. It was because lookups in the JVM were atrociously slow, requiring a native layer to get out of the Java sandbox.

Speculating here, I suspect that an important Sun or IBM customer had an app that did lots of DNS lookups and the performance stunk. So, an engineer did a quick 'fix' to cache DNS lookups. Customer was happy, everyone moved on. Some time later this quick fix got ported into the mainline code base. But, it appears that nobody did a proper analysis of this quick fix, ie. respect TTL on DNS. Maybe supporting TTL wasn't important because this was back in the early days of Java when it was trying to win the desktop war and desktop apps weren't really expected to be long lived processes.

Personally, I don't think you should use DNS at all for intra-network service communication. I think the naming scheme described in the article is useful for humans, but for services trying to connect to other services, I would stick with hardcoding or Zookeeper.

Cnames are really quite poor for this unless the cname information is being driven from some other tool that you can easily do lookups on.

Otherwise you won't be able to get that classifying information on the end host.

Using a deterministic ID for machines is definitely the way to go.

We actually studied this for a while and MAC address is a pretty universal ID that works virtually anywhere. We use the smallest physical non-zero MAC address in case of multiple NICs. We considered using chassis or baseboard serial numbers but it gets too vendor specific.

The problem you run into with using the MAC address is what happens when you swap out the NIC?

You use the new one? Either that or dont. Doesn't really matter that should all be automated shouldn't it?

I'm not following. The whole purpose of naming is to make it easy. I don't want to remember 000a43dbac72.example.com, it's easier to remember "pine" and "oak" and trust me, after working in the environment a while, everyone knows what every keyword is.

Over a certain size you want your tooling to take care of that knowledge not you. You don't want your sysadmins or developers deriving information from records that can change.

if your developer or sysadmin assumes that the server named castle will always be something special instead of looking in the CMDB or other ENC for that information you will have less fun in the long run.

tl;dr random names with a central ENC forces you to only get meaning or facts from one central repo.

The benefit here is mostly just with larger infrastructures where you're gonna run out of wood names and where agreeing on and enforcing naming standards is a futile and painful process.

With logical mappings, you just use chef/puppet/cmdb to do something like sshnode prod app 1 to connect to the first production app node or sshnode prod app to connect to all the production app nodes. CNAMES can do this too but then you run into potential DNS consistency issues. With this you get the human friendly abstraction and keep the machine friendly determinism.

Of course when your chef server goes down this can be a PITA (I recommend building in some knife search caching!)

This seems reasonable. I think the 'themed' naming works well for up to perhaps a 100 or so servers/devices managed by a single team, but with tons of automated provisioning, I suppose unfun naming is a requirement.

I am still trying to understand their reasoning myself, unless you just have hundreds of servers I can see no reason to resort to numeric naming methods, that might be fine for arrays/sorts/computers but not people. It looks good on racks but for representation to humans, argh.

Ours are named for location and function. There are some large servers which have six character names as they so unique/important to the organization there is never a question. Usually though many fall into <owner+location+environment)

However I can see justifications made in either direction, its no worse than coding conventions that people come up with.

That sounds like premature optimization for a lot of people.

Part of the problem is that hostnames are so quickly ingrained into our workflows that when you only have a few servers there's no point, and when you have a bunch it's too late – unless you want to spend a lot of time changing every reference.

If you start building a small cluster which you know is going to grow into a big cluster, then these aren't really premature optimizations so much as laying a proper foundation. It's the difference between rewriting some of your Python in C and making sure all your python code is packaged into sensible objects and grouped into libraries.

Reply on news.ycombinator.com