5 out of 7 .io nameservers appear to be down | Hacker News Reader

5 out of 7 .io nameservers appear to be down

pulse.turbobytes.com

109 pointssajal8365 comments

https://gigaom.com/2014/06/30/the-dark-side-of-io-how-the-u-...

"The rights for selling .io domains are held by a British company called Internet Computer Bureau (ICB), [...] The British government granted these rights to ICB chief Paul Kane back in the 1990s. ICB gets to run .io “more or less indefinitely, unless we make a technical mistake,” Kane told me. (ICB has so far run a stable .io namespace. It should be noted that Kane is a respected veteran of the infrastructure scene, and has been entrusted by ICANN with one of the 7 so-called “keys to the internet”.)"

Ooops...

So, this mistake might cost him the rights?

What's the mistake?

The reason there are multiple DNS servers is in case one/some of them have problems. There are two other root servers for the .io. zone that are apparently functioning just fine. That means the overall system is working as intended, no?

Also, I don't think we know (yet) why five of the seven are down. If it turns out to be some "amateur hour" mistake then, sure, I could see it being used against ICB. If, however, the underlying issue is/was out of their control, then why should they be penalized?

ETA: It appears that the name servers are actually "up". They respond to ICMP echo requests but aren't answering queries:

  $ ping -q -c 5 a.nic.io
  PING a.nic.io (64.251.31.179): 56 data bytes

  --- a.nic.io ping statistics ---
  5 packets transmitted, 5 packets received, 0.0% packet loss
  round-trip min/avg/max/stddev = 45.123/45.283/45.453/0.106 ms

  $ dig ns docker.io @a.nic.io

  ; <<>> DiG 9.10.2 <<>> ns docker.io @a.nic.io
  ;; global options: +cmd
  ;; connection timed out; no servers could be reached

To be fair, even if it was "amateur hour", it was apparently mitigated by practices which kept two of the seven up.

I've watched other companies do far worse.

Try using +norec in dig and you'll get a response.

Interesting. I was using "+trace" earlier (which implies "+norec", IIRC) and still wasn't getting a response from most of them.

Interesting, it works locally for me when using +norec (i.e. Recursion Desired = false)... but is there a valid reason for an authoritative to to respond anything if RD is set to true? FYI Pulse always sets RD=true

For those of us not up to date with the wheelings and dealings of organizations such as ICANN, what are these 7 "keys to the Internet?"

DNSSEC root key shares.

DNSSEC root keys (or a portion thereof)

The "keys to the Internet" is a misnomer. He is a "recovery key shareholder", which means he holds a smart card in an m-of-n configuration that allows ICANN to decrypt a backup of the Root Zone Key Signing Key in a disaster recovery scenario should the KSK need to be rebuilt.

The primary roles are the cryptographic officers, these are Internet community members who attend key signing ceremonies to observe use of the KSK.

Here are a list of the trusted community representatives: https://www.iana.org/dnssec/tcrs

> The primary roles are the cryptographic officers, these are Internet community members who attend key signing ceremonies to observe use of the KSK.

While I realize those key signing ceremonies are probably just a bunch of people sitting around playing games on their phones, waiting around for their turn to give their keys to someone who's typing in all of the appropriate commands, I want to believe they're in some sort of dungeon, wearing black, hooded robes and chanting Gregorian chants while doing it.

I dunno, maybe I'm just weird.

It's a little more interesting than your guess, but not quite as cloak and dagger as it could be.

Here's a good article about the ceremony itself: http://www.theguardian.com/technology/2014/feb/28/seven-peop...

"The dark side of .io: How the U.K. is making web domain profits from a shady Cold War land deal"

https://gigaom.com/2014/06/30/the-dark-side-of-io-how-the-u-...

I think I trust even an ICB root server more than I'd trust one running on that island

That's so stupid. If some islanders had their own gTLD, it wouldn't be .io. It's not like it's some natural right that belongs to the island or the peoples. Chagos Island? So they'd have .cg or something like that. Not so valuable eh?

They actually seem to work fine but you have to set the non-recursive bit.

  $ dig ns io @b.nic.ac +short

  ; <<>> DiG 9.8.3-P1 <<>> ns io @b.nic.ac +short
  ;; global options: +cmd
  ;; connection timed out; no servers could be reached
  $ dig ns io @b.nic.ac +short +norec
  b.ns13.net.
  [... extracted for brevity]
  ns3.icb.co.uk.

This is the correct answer. The monitoring tool is sending the wrong kind of DNS queries to these servers, and these servers are filtering them out. You can argue the servers should at least respond with an error, however they are functioning correctly when you ask them for an authoritative response (i.e. a delegation)

(monitoring tool author here) Agree. Makes sense. We should have run this test with RD bit unset.

But .. IIRC if RD is set when quering a non-recursive, it should respond normally with authoritative response Recursion Available (RA) flag unset. It does not mean it should drop the query totally.

Dropping bad queries is often a DDoS countermeasure.

This is indeed a typical configuration for authoritative name servers.

Is there a table of TLD's ranked by nameserver reliability? It seems like a useful thing for those of considering something more exotic

I don't personally know of one (which isn't saying much) but that sounds like an interesting thing to monitor. Maybe an HN'er who has written their own monitoring system (and provides it as a paid service) could begin monitoring queries/responses from the many TLD root name servers.

"i/o timeout"

That's ironic.

What if the actual indian ocean timed out? That'd suck.

From your previous comment complaining about HN:

"The internet doesn't need another reddit, it's bad enough as it is."

Yet you insist on making comments that only belong on reddit. Can I convince you to delete your account?

Says the loser with a reddit account

Fair enough if you don't appreciate the comment, but asking someone to delete their account is incredibly rude. There are much more tactful ways of telling someone their attempt at humour is not appreciated.

There's a good 2004 documentary by famous film maker John Pilger on the topic of Chagos: "Stealing A Nation".

Fully available on youtube, about 1h long: https://www.youtube.com/watch?v=0zhGvId4fcc

It seems that now also startup companies are (unwillingly, I presume) contributing to this lasting injustice.

5 out of 7 have been down for ages, literally ages.

How do you know?

Not sure if this is what OP is getting at but I do believe some of these .io domains are failed projects that are no longer pointing to an active web server.

These are tld servers, not domains. TLD server are root dns servers, acting as source of truth for the domain names.

No, a root server is shorthand for the authoritative name servers that serve the root zone of the DNS. Here is the list: http://www.iana.org/domains/root/servers

Servers configured to be a source of truth are called "authoritative name servers".

There is no technical distinction between "TLDs" and "domains", they are all just domains, they are just different levels of the DNS hierarchy.

Did some research myself a while ago, no source other than my word I'm afraid.

https://pulse.turbobytes.com/results/5595115aecbe40693500017... better link to show that its not a local resolver issue.

... and the two remaining nameservers look to be run by the same organization; the /16 that they both reside in (49.212/16) is being announced by "Sakura Internet" (AS9371).

It looks like they do have multiple upstreams, though.

My .io has been up and down about 25 times in the last 15 hours. The downtime has been about 1.0 - 2.5 minutes each occasion, with a down period of 1 hour 48 minutes at the start.

The latter downtimes look planned, as they are for exact periods (1 minute 0 seconds, 1 minute 30 seconds, 2 minutes 30 seconds, etc)

Here's an extract from the log with timing incase anyone's interested: https://gist.github.com/anonymous/ed37826bc66c23d6c791

What are you using for monitoring / what generated these logs?

Unless your monitor is attempting a connection every, e.g., one second, you're going to end up with the "exact periods" you describe (i.e., presumably your monitor is only attempting to connect once every 30s).

Also, this would be more indicative of a problem with your web host, not the .io root name servers, assuming the TTLs on your RRs are set to > 30s.

The message format looks like something from UptimeRobot.

I was already slightly concerned about using getting a .cc domain for my email address, due to it being a 'lesser' TLD and thinking this sort of thing might happen. Are there ways to mitigate this risk in general, or is it something we generally have to accept if we're not choosing .com domains?

i can just see the option to avoid TLDs which "belong" to companies acting in volatile markets... which may be indeed of concern with all those fancy new generic TLDs.

i would welcome better proposals, too.

I think (but don't hold me to it), that country-ran (eg, .co.uk, .us) or major TLDs are safe (eg, .net, org)

Well, happens with .se. in the past http://securityandthe.net/2009/10/13/se-domains-offline/ and other tlds as well.

I don't understand what you mean by "country-ran." .us and .uk are ccTLDs, meaning they were created and recognized 2-3 decades ago by ICANN based on ISO country and Territory codes. ccTLSs are among the oldest continually operating TLDs. In fact, .io is a ccTLD.

But that doesn't make them "run by a country". .us is not run by the US government. It is run by Neustar. There is not something that says Neustar won't have poorly available/slow name servers, beyond that same statement that the .io people said.

I was under the impression that ccTLDs are administered/ran by the country they belong to (eg, I thought .us was administered by the US government). It sounds like I'm wrong though, my bad.

Countries have the oversight, but they usually contract out its administration to some company.

.com would also be a very safe bet.

I'd say that depends heavily on what TLD you are talking about. I find IIS[1] (the foundation that runs .se) freaking awesome, but I guess .se wouldn't make much sense if you're not Swedish.

[1] https://www.iis.se/english/about-se/

Whatever domain you choose, you're reliant on the root servers for that particular TLD. All you can do is look at who runs it and see how competent you think they are (well, I guess you could try to negotiate an SLA in your contract with them)? Just as with e.g. national telecom operators, or national postal services, there are nations (large or small) that have highly competent, reliable TLD administrators, and there are nations that have unreliable ones.

The DNS for .cc is run by Verisign, who also run DNS for .com and .net. I wouldn't worry about .cc when it comes to namserver reliability.

I've had a .CC domain for the last 6 years or so. Got it originally for starting a computer consultancy (hence going with .CC instead of traditional .COM). I folded the operation but kept the domain and have been using email based on that domain as my personal email for all these years without disruption. (A caveat though: email is based on Google Apps for your domain not sure if that matters.) I went with it because there was a decent-enough (meh) name behind it (eNom). If the backers of the domain had been some unknown organization I'll be honest, not so sure i would have used it. I can't provide fool-proof ways to mitigate issues, but for what its worth, I've had luck with .CC all these years.

The good of domains like .CC: * Your desired domain name likely to be more available than .COM/.NET/.ORG. * If a tech-centric business, your clients might think of you as edgy and non-conventional. * Some domains like .CC have one less character to type - ok, so its a small benefit. ;-)

The bad of domains like .CC: * Many non-tech folks still think there are only 3 or 4 TLDs in the world...so you have to specificy something like: "It's .CC not .COM"...and they usually respond with "Huh??" Its funny, in all the years I've had my domain, the only reason I've not received email is because of folks mis-typing ".COM" instaed of .CC. * Filling out forms on websites whose devs mistakenly assume - since your address does not end in .COM/.NET/.ORG - that it must be an "invalid email address". Though this occurs very, very rarely now-adays.

Am I the only one who noticed the mustache templates in <title>-tag while the page was loading?

Thats angularjs template... We need to put in some placeholder there.

using <title data-ng-bind="pagetitle"></title> should eliminate the flicker

just did that. thanks

Alternately, ngCloak:

https://docs.angularjs.org/api/ng/directive/ngCloak

And yet the internet keeps working.. :) Docker.io still resolves for me, and i guess many others do?

every y out of x times a DNS recursor will fail to resolve .io .... ratio of y:x is based on complex srtt algorithms, and traffic levels. Not a big issue for popular hostnames like docker.io because most recursives would have the delegation in their cache.

Caching is great isn't it.

But a .io domain sure hip! Can't go with something boring or traditional like .com, .net or .org ;)

That's because anything halfway human readable is taken under those TLDs.

There are plenty of human readable domains available, they're just long.

I hope this is costing them lots of money for fucking up.

We seem to be POPing : http:/popteam.io

Reply on news.ycombinator.com