Sinkholed: A DNS Horror Story (2019) | Hacker News Reader

Sinkholed: A DNS Horror Story (2019)

susam.net

127 pointssusam37 comments

> Unfortunately, their operation inadvertently flagged my domain name as one of the domain names to be sinkholed because it matched the pattern of command and control (C2) domain names generated by a malware family named Nymaim, one of the malware families hosted on Avalanche. Although, they had validity checks to avoid sinkholing false-positives, my domain name unfortunately slipped through those checks. [...]

That's incredible. That "[the domainname] matched the pattern of command and control (C2) domain names generated by a malware family named Nymaim" was enough to get it sinkholed is nuts. There should be a fair bit of manual checks here before applying this sort of death penalty.

Take my username. I actually got targeted by my bank some years ago for a similar reason: "crypto" surely means you're doing crypto-currencies, right?, so you must register as a money services business (MSB)! Uh, no, nothing of the sort, and thankfully I was able to disabuse them of the notion that I had anything to do with crypto-currencies. (Indeed, I'm vehemently opposed to proof-of-work currencies, and as for proof-of-stake, why not just do double-spend detection and leave it at that?)

I've used a domain with "uber" in it for over 20 years, and since the ride sharing service started I have quite a few issues pop up. At least people know how to spell it now.

Yeah, how do you spell it?

Über, with an umlaut like the original German word? Or do you spell it Ueber, where the "ue" stands in for an umlaut in character sets which are incapable of reproducing the glyph?

Or do you spell it Uber like the American ride-sharing service, ignoring niceties of pronunciation and trading those for convenience by people who only speak English and don't know German phonology?

As someone who has a name containing a non-ascii letter living in the US, I greatly prefer simply removing the dots or apostrophe or whatever and just use the ascii version. Those transliterations like ü->ue or ø->oe cause much more problems than they solve. Yes, there are lots of letters for where this doesn't work at all, but for the ones that do, the principle of least surprise is better than phonetical correctness, imo. I pick my battles with Anglo- and US-centrism and this one isn’t worth fighting for, imo. “Whatever the letter looks closest to” has a much higher chance of a consistent interpretation. This isn’t even the fault of anglocentrism – it’s an issue between any languages with different alphabets. Heck, I don’t know Cyrillic, but at least with say Vietnamese i can go from ê->e without even turning on my brain.

EDIT: oh, and reconstructing a word or sentence with missing umlauts etc is very easy to do, at least in my language. You can infer by context if there are ambiguities in 99% of cases.

The word Uber has existed in English speaking countries for much longer than the ride sharing service. Using that spelling doesn’t affiliate you with the company. The opposite actually as the company used a common word.

Arguably, Uber is different from uber, meaning outstanding or supreme, and the former has only existed since 2011 due to a complaint from San Francisco taxis causing a rename from Ubercab.

My impression was that, when botnet sinkholing was first practiced, a lot of care was taken over avoiding collateral damage, but if so, then the standards seem to have decayed.

This might have been a consequence of frustration - effective action seeming to be just out of reach - but I have also seen the fallacy that goes "we are the good guys, so therefore what we do is for the good." See also "normalization of deviance."

This is why I suspect that, if the AI Apocalypse is technically possible, it is inevitable.

Like anything else, when it’s new, the A-team people are working on it. When changes slow and products or services move to steady state, care and feeding moves to dollar focused app teams.

This is actually terrifying to me. If my domain goes, so goes my email -- everything is @myname.com, so I won't receive any email. If my email is gone then I just instantly lost access to a big chunk of the internet services I use, and my online identity might as well be dead. Sure I still have my work email but that's no consolation for all the services registered to my domain email.

I'm not sure what the right course of action here is to mitigate such a risk. Add backup emails on a different domain registered with a different registrar to services that support such a thing? Stop worrying because it's rare enough that it's a waste of time? Is this what happens to people that get banned on gmail?

Yes, different domain on a different registrar is the usual solution.

I've recently been through this and it is a total nightmare. I lost my phone number too, so there was no way to use that validation channel. And many systems have no human support now, so you are super screwed.

And then things break for no reason at all and panic the shit out of me. My Namecheap TOTP stopped working today for no reason at all. So I had to go through hours of support to verify my identity, but the whole way through I was thinking "What if I can't verify?"

It's a rabbit hole to go down, for sure.

I spent a while one week going through all my online identity, services I use, etc and putting together basically a dependency tree.

What kicked this off was an overlapping but probably more unusual concern. Basically, I was worried that I had things _too_ secure. If I lost certain access, I'd be _screwed_. It's great that this account needs 45 factor 6 dimensional holographic password authentication to log in, but what happens if I lose something or get bonked on the head and forget something or... how can I recover access, but also set this up in a way where the backdoor I leave is _not_ one that's easily accessible to others.

Anyway, long story short is I have a separate ccTLD domain from my country that is _exclusively_ for use as the root of my identity/recovery. Everything related to it is in a separate account. It charges to a credit card that's not used for anything else. The only thing it does is receives email and dumps them into object storage so I can periodically review. (I don't want them forwarded elsewhere in case the email is something like a password recovery email.)

The recovery solution for this is the ccTLD's dispute resolution policy, and finally my (local) courts. As sexy as that Cocos Islands or Indian Ocean Territory or vanity TLD is, I have a lot more options more easily available to me with my local ccTLD administrator and local courts. I'm pretty much relying on the court's ability to accurately verify my identity as the lock on the back door.

> The recovery solution for this is the ccTLD's dispute resolution policy, and finally my (local) courts. As sexy as that Cocos Islands or Indian Ocean Territory or vanity TLD is, I have a lot more options more easily available to me with my local ccTLD administrator and local courts. I'm pretty much relying on the court's ability to accurately verify my identity as the lock on the back door.

I work at a domain name registrar, and I agree completely. In fact, I have recommended this attitude for years¹. Use your local ccTLD if at all reasonable.

1. <https://news.ycombinator.com/item?id=32802909>

<https://news.ycombinator.com/item?id=32801880>

<https://news.ycombinator.com/item?id=30598899>

Maybe use a secret sharing scheme and give parts of the secret to a few people you trust?

I'm using my Hotmail (yes I'm old, shut up) for my critical-to-life email communiques because I'm reasonably sure Microsoft won't randomly shit on me, unlike Google whom I'm always wary of even though I've had my Gmail for the better part of 2 decades now.

For services that allow tying more than one email to, I've got one or the other as a backup/secondary so I won't be completely fucked one way or the other.

As a general concern it's good to not have all your eggs in one basket anyway. If it's critical, have a backup plan and ideally a backup plan to the backup plan.

I have exactly two email identities and one mobile phone number, oh and a slack handful of "POTS" (VOIP really) numbers. I run the lot myself.

VoIP - my company has a retail account with a wholesaler. I can allocate UK phone numbers at will. My wife set up a new business a few years ago and I had another phone number with distinctive ring running within 20 mins. Obviously not your usual setup. My dad and stepmum have a RasPi running their phones at home. At my place it's a VM.

email - despite the usual HN groupthink with regards email, I have been running tiny email systems as well as some rather larger ones for quite some time - decades. I do note that this does not work for everyone but for me in the UK it does. I do have access to long running static IPs and have made sure that their "reputation" is top notch.

I recently added a "vanity" domain for a friend so they can escape ISP lock in. Their surname is the same as a village in North England and some wankers had squatted on village.co.uk and wanted £25,000 for it. I advised my friend that if he registered a UK LTD with his family name then we could probably get the .co.uk by default. I registered .org.uk instead - too much faff for my friend. I set up SPF to use my family's MTA (Exim) to ensure that things tie up. A .org.uk domain costs about £6 + VAT (sales tax) hereabouts. I suggested £20 per annum as a charitable donation of his choice, in return for me running his email. He chose St Margaret's Hospice. It turns out that both my mum and his mum received end of life care from St Madge's.

I have digressed somewhat!

You do need at least two identities for safety. You mention "my domain" - do you actually run it? If not then you need another one as well. Do not, whatever you do, rely on any single hyperscaler to give a shit. The well informed user can route around damage in the cloud. It sounds like you are unlikely to run your own system(s) so have a backup plan. Remember that say FB provide a smtp service if that helps - ISTR that id@facebook.com should work. There are lots of other freeish email offerings. Just make sure you have two well known identities that those whom you care about can contact you with. You can also recommend the same advice to them.

How the fuck does some random-ass company have the authority to just yank domains away instantly like this, without consent or confirmation? That's what I want to know.

So, nothing to do with the actual DNS, but instead with the domain name registrar.

Nothing to do with DNS? Just something to do with the organization that controls some DNS records

At least, not a technical problem, and not even with the zone for their domain, but with the zone of the .in domain.

systems include people

Reminder: you can't really own a domain name.

Use public keys instead. Like i2p and tor do.

Unfortunately the rest of the world uses DNS

I like to be able to actually tell someone my domain name/email address/etc., verbally.

Rattling off a 56-character base32 string just doesn't work.

Fine, as long as you understand that you don't own any of those pronounceable identifiers.

https://en.wikipedia.org/wiki/Zooko's_triangle

What do you mean: "you can't really own a domain name."

That seems to be like saying you can't own a house! Unless you run a registrar, you have to engage the services of a registrar to register your domain with the internet infrastructure. In the UK, the Land Registry is where you register your claim for a particular plot of land and property ownership and where your mortgage provider registers an interest against your property until it is paid off. I forget the precise legal terms.

The internets does have a reasonably thorough concept of domain ownership and there is a decent set of precedents. eg MacDonalds wrested control of macdonalds.co.uk from a Scottish shyster who has the surname "MacDonald". He did not have a Ltd company in his name but Maccy D does.

The difference is that the postal service will still forward your mail for you if you sell or lose your house. There is no way to make that happen with a domain name. A person who registers your expired domain name not only has no obligation to forward your email, they can also just read it and potentially get access to any services you set up using that email address as authentication.

Just as an aside - the UK land registry isn't a perfect system at all, even where land is actually registered. For example - I own a flat where the land registry scan of the original long lease which legally documents what I own is incomplete. They don't keep original documents any more since they scanned everything and got rid of the originals years ago to save storage costs. Neither I or the freeholder have original copies either. Theoretically they are supposed to pay to correct this if I complain - but part of what's missing is a plan showing the extent of what I own, which opens up a potential dispute with the freeholder as to what land the lease was originally granted over and I don't really want to have a legal fight over it (and I'm not sure the land registry would cover any legal costs of a dispute). It's not a day to day issue but there's some risk there for the future.

"the UK land registry isn't a perfect system at all"

No shit! I was an Army brat and lived in what was called West Germany. It may have been a state thing (Nord Rhein OP) rather than a federal thing but houses often had little markers, similar to surveying benchmarks to demarcate property boundaries.

In little Britain we have err the Land Registry. It's not quite as bad as you say. Since around 2007-2010 (I can't remember exactly when) plans submitted to Planning had to have a lot more detail added. For example you have to include trees and hedges but not co-ords of the boundary to 5mm from the nearest OBM. I can personally survey better than that and I'm just a Civ Eng graduate who ended up pissing around with computers.

My house came with its original and subsequent conveyancing docs. It was built in the 1920s, so quite modern. The farmer who flogged off a bit of land, near to a railway line also provided a mortgage/loan to the woman who bought it. Its been through a few hands since. One of the subsequent owners lives up the road. He and his wife (she passed away recently) divided the plot, another house was built and a double garage linking the houses was added.

Anyway, the Land Registry is quite loose but it works. It seems that Little Britain is quite content on a microscopic scale. We somehow manage to not get too whizzed up over boundaries too often. The boundaries somehow manage to manage themselves.

Isn't there a problem with key-as-identity is that if you lose control for whatever reason, you have no recourse?

If you're really so worried about that, escrow your key with an undeletable backup service like the one provided by rsync.net. These are quite popular as ransomware protection.

https://www.rsync.net/resources/howto/snapshots.html

Frankly, this concern is massively overblown by people looking for an excuse to replace cryptography with government identity papers.

I would really like to know what sort of traffic patterns that caused the detection. How would you accidentally mistake susam.in for a C2 server? Where are they sniffing traffic at to determine this?

This and the "how the fuck does this happen?" comment...

There are two technical components which are salient to the Avalanche story here which are DNS related.

First is <<fast flux>>. A rough sketch of this looks like servers behind reverse proxy servers, and DNS servers. Both the proxy and DNS server components are intended to cycle extremely fast, rendering IP address blocking impractical and tracking difficult. We'd often find that the actual mothership servers used named virtual hosting and by sending different Host: headers we could tickle different phish.

Second is <<DGA>> or a Domain Generation Algorithm, the point of which is to enable autonomous discovery of Command and Control servers by implants. In practice you can think of a DGA as TOTP for domain names. So at any given time the algorithm offers a bag of possible C2 server domain names (which constantly changes) and the implants rattles doorknobs until it hears "notary sojack" from one of them. (There can be lots of these, thousands or tens of thousands at any time.)

I think DGA is the concern here. You may think there's some traffic sniffing going on, but that's only the case for observed implant activity.

Let's look at some implant and its C2 from a traffic standpoint. The implant is going around asking "notary sojack? notary sojack?" to every Tom, Dick and Carol and even to cats and garbage cans. The C2 looks like a garbage can but has an incongruous collection of fans who are also observed muttering "notary sojack" at random objects.

It was possible in most cases to reverse engineer the DGA and seeds with confidence. DNS zones would be populated with the flavor of the day (and maybe the days prior and after) and utilized by caching resolver operators as response policy zones (RPZs) or e.g. Bro lookup tables. The notion being if something in your org (using your resolvers) is doing DNS lookups for a bunch of these, you might want to take a look and see if it's rattling doorknobs and muttering "notary sojack". That's the idea. Technically, somebody could block on it but there is the risk of false positives. Your vendor might vett the list in some fashion; or not.

I don't recall domains being seized as commonplace in my time. I could make up hypotheticals as to what might cause this to occur but I've got no evidence or experience supporting one over another. It could've been a mistake.

In any case, after a few days the DGA moves on and the domain is useless; that comports with the domain being returned to the owner.

(I'm not under any noncompetes at this time.)

Hmm, so then susam.in supposedly came out of a DGA in the wild being observed (or was a reverse engineered possibility) and then matched some level of checks to try to weed out false positives.

Still feels sloppy that their whatever they're hosting looked enough like a C2 server/reverse proxy point to a C2 server to get their domain _seized_. That's quite a drastic action, especially across countries being it's a .in domain.

99% correct in my estimation. Sloppy... that's where I'm not sure; heavy-handed? Certainly. I said I wasn't covered by noncompetes any longer, I didn't say I felt comfortable spilling all on HN. I'm also not current on what covenants are in place around .in specifically.

If I was investigating this I would start with the hosting for susam.in and could it have been compromised.

If threat intelligence intrigues you, you're welcome to stalk me and send me a connection request on LinkedIn. And as a footnote to the "how the fuck..." comment, maybe it should be FCANN rather than ICANN for truth in advertising.

This is why you don't use meme TLDs.

Reply on news.ycombinator.com