Tell HN: All .fj domains have gone offline | Hacker News Reader

Tell HN: All .fj domains have gone offline

93 pointsmarkdown49 comments

domains.fj

fiji.gov.fj

usp.ac.fj

etc.

http://www.iana.org/domains/root/db/fj.html

View on HN

University of the South Pacific, which runs the DNS servers, [1] is up.

Google's DNS server at 8.8.8.8 is having problems.

Lookup using DNS server of sonic.net, which is a normal DNS server.

    nslookup usp.ac.fj
    Server:  127.0.0.53
    Address: 127.0.0.53#53

    Non-authoritative answer:
    Name: usp.ac.fj
    Address: 144.120.141.80

Lookup with Google DNS:

    nslookup 
    > server 8.8.8.8
    Default server: 8.8.8.8
    Address: 8.8.8.8#53
    > usp.ac.fj
    Server: 8.8.8.8
    Address: 8.8.8.8#53

    ** server can't find usp.ac.fj: SERVFAIL

Checking example.com to make sure Google DNS server is live.

    > example.com
    Server:  8.8.8.8
    Address: 8.8.8.8#53

    Non-authoritative answer:
    Name: example.com
    Address: 93.184.216.34
    Name: example.com
    Address: 2606:2800:220:1:248:1893:25c8:1946

Also fails with 1.1.1.1 DNS server.

The Fiji DNS servers are unreachable, but you can get as far as a router at the University of the South Pacific. So the problem is local to there.

    traceroute 144.120.146.1
    traceroute to 144.120.146.1 (144.120.146.1), 30 hops max, 60 byte packets
    ...
     8  AARNET-PTY.edge5.London1.Level3.net (217.163.113.74)  274.795 ms  274.476 ms  273.781 ms
     9  xe-0-0-0.pe1.a.suv.aarnet.net.au (113.197.15.213)  312.534 ms  311.764 ms  311.766 ms
    10  fastethernet0-0.aarnie.usp.ac.fj (202.158.204.194)  313.356 ms  312.176 ms  312.363 ms
    11  * * *
    12  * * *

[1] https://www.usp.ac.fj/

> University of the South Pacific, which runs the DNS servers, [1] is up.

Not up from within Fiji. All government websites (and email) are down. Banks, Universities, mobile money wallets, etc.

DNS for Fiji seems to be back up.

Yeah, came back online this morning. Was down for around 18hrs I reckon.

That #10 step isn't on Fiji, Fiji isn't 1ms away from .au

xe-0-0-0.pe1.a.suv.aarnet.net.au isn't in Australia

Oh, it's the far end of their circuit? Okay, that makes more sense.

Oddly enough, right now..:

> server 144.120.141.80 Default server: 144.120.141.80 Address: 144.120.141.80#53 > usp.ac.fj Server: 144.120.141.80 Address: 144.120.141.80#53

* server can't find usp.ac.fj: SERVFAIL

Fijivillage.com reports: "All websites and apps hosted in Fiji with the dotcom.fj suffix are currently down and this has also affected Vodafone’s M-PAiSA services.

This is due to an outage in the University of the South Pacific hosted dotcom.fj domain." [1][2]

[1] https://www.fijivillage.com/news/All-websites--apps-in-Fiji-...

[2] https://twitter.com/fijivillage/status/1501070675691278339

And this folks is one of the reasons you don't use a 1 minute TTL on your DNS responses...

shamefully opens domain registrar and changes TTL

15 min?

I would recommend an hour for almost everything except where very fast updates is expected, in which case 5m is my lowest number (I work at a registrar).

Obviously registrars allow 1m - is there a reason not too besides extra load on DNS servers?

If the DNS server is out for more than 1 minute you'll get an outage. You also increase latency by preventing anybody from caching the response for more than 1 minute.

There are rumors of DNS resolvers deciding that some TTLs are “too low” to be valid, and applying their own default TTL value instead, thereby negating any benefit a low TTL would have had.

I use 24 hours on anything I'm not planning on making changes to. High TTL is a better experience for customers because they'll probably have it in local-ish caches even if there is internet routing disruption.

And 10 mins on anything I'm about to make changes to. That means if I accidentally make the wrong change, the 'blast radius' is minimized.

Obviously, when changing 24h down to 10 mins, keep a close eye on DNS server load, packet loss on links close to it, etc. If in doubt, raise and lower ttl's slowly.

3 of the 5 nameservers for fj are online and responding (they are all anycasted around the world). It looks like connectivity to Fiji itself is what is disrupted.

Two are operated by gransy.com and one by PCH.net (which you should consider donating to and helping random country codes stay online: https://www.pch.net/about/donate).

It's not connectivity to Fiji. The internet works just fine across the country. Only .fj domains (and email accounts) are affected.

This might be a DNSSEC issue rather than a nameserver reachability issue:

https://dnsviz.net/d/fj/dnssec/

I guess Google's DNS rigidly adheres to DNSSEC.

https://developers.google.com/speed/public-dns/faq#dnssec

"Google Public DNS is a validating, security-aware resolver. All responses from DNSSEC signed zones are validated unless clients explicitly set the CD flag in DNS requests to disable the validation."

Looks like you were right: https://lists.dns-oarc.net/pipermail/dns-operations/2022-Mar...

It's true...

  couldn't get address for 'ns4.fj': not found
  couldn't get address for 'ns5.fj': not found
  couldn't get address for 'ns2.fj': not found
  couldn't get address for 'ns1.fj': not found
  couldn't get address for 'ns3.fj': not found
  dig: couldn't get address for 'ns4.fj': no more

Using a Recursive DNS server its down for me. UK based. I also setup a RIPE measurement for those interested. It asks for the NS record of ns1.fj.

https://atlas.ripe.net/measurements/39252885/#general

Is there anyone here in Fiji who can say they are OK?

Reachable for me.

Would you mind saying where in the world you are?

I'm late to the party, but https://fiji.gov.fj/ worked for me, although it was rather slow to load.

I'm in England, for what that's worth.

Working slowly from Germany as well.

Not working for me, in New Zealand.

This is why I always say you should think twice when choosing a ccTLD if you don’t have a presence there. Not every registry has the same uptime as Verisign.

And yet to my knowledge this has never happened before.

Didn't .io ( which is a crime against humanity to boot) go down for a few hours a few years ago?

No idea. I was referring to .fj

Why do you say .io is a crime against humanity?

.io is the TLD for the British Indian Ocean Territory, which is associated with the forced displacement of the Chagos islander people to make room for the Diego Garcia military base, amongst others. The British unilaterally decided to detach the Chagos archipelago from Mauritius before it got independent, which the deported people say (not without reason) is a crime against their human rights.

To back that up - the Chagosian claim is backed by the UK and rulings at the International Court of Justice. The UN supports Mauritius' claim to the territory.

Even besides the literal crimes against humanity, the .io domain really doesn't need to exist - what with the British Indian Ocean Territory having no permanent population and really not being a country at all.

"Being a country" isn't the criteria for the ISO code or domain.

Consider AQ, TW, GG, SJ, EH, HK and many others.

TW is a controversial inclusion in your list.

Several of them are controversial to someone.

AQ is controversial to countries who want to exploit Antarctica's natural resources.

TW is controversial to China.

GG is controversial where it's used for tax evasion etc.

EH to Morocco, HK to the UK/China.

Yes. https://hackernoon.com/stop-using-io-domain-names-for-produc...

Also there were some comments from the owner at the time that I thought were pretty unprofessional about it, too.

Maybe not, but it is predicable that the quality and quantity of resources and governance available to a registry has an impact on their operations.

I vividly remember .de being offline causing trouble for a couple of hours in 2010.

https://www.theregister.com/2010/05/12/germany_top_level_dom...

It's too late to edit my comment, but I was referring to .fj, not ccTLD's in general.

Given the amount of past outages of ccTLDs we've seen (almost none), i'd say this problem is negligible and nothing to worry about.

There is one very recent exception however, the russian plans to partly separate from the internet shortly (especially regarding DNS) could well be an issue.

Technical and legal issues that cause issues for customers at small ccTLDs are not uncommon. It all depends on who is running the registry. Try "cctld hijacked" and "cctld downtime" in your favorite search engine for a dozen or so examples.

Registries range in resources from "highly experienced billion-dollar organizations", to literally "we have a guy who updates the zone file in notepad", and they range in legal environment from, "functional democratic state with well-established judiciary" to "our government was violently overthrown this week".

Whether or not the problem is negligible depends on your particular use-case. For some, it might be a good choice, for others, it might not be.

Like I have said before¹²: Neither your registrar nor your chosen TLD registry should be in the habit of suspending domains at the drop of a hat, or be at risk of going out of business suddenly. If you mostly trust your local government, your national ccTLD should suffice. In fact, it should be your default choice unless you have strong indications it does not fulfill the above criteria.

1. https://news.ycombinator.com/item?id=29108933

2. https://news.ycombinator.com/item?id=21614298

Reply on news.ycombinator.com