Sinkholed
susam.in
susam.in
This blog post summarizes the timeline and the events that occurred to resolve the domain transfer issue. Like I have mentioned in this blog post, multiple parties such as Namecheap Support, the Shadowsecurity Foundation, and NIXI helped me in resolving this issue. Thanks to all of them and special thanks to Namecheap CEO, Richard Kirkendall, for looking into this issue on priority as soon as he became aware of it.
I wish the domain name management, in particular, and the Internet, in general, had a higher degree or decentralization, so that it were technically impossible for something like this to happen but I think there are many factors at play that are currently preventing from such an Internet to become mainstream at this time.
Have you consulted a lawyer?
It seems that the Shadowsecurity Foundation did act recklessly. But you'd need to prove monetary damages.
But perhaps they'd settle to avoid the hassle.
Edit: This is an admission of guilt:
> He explained in his email that my domain name was sinkholed accidentally as part of their Avalanche operation.
In fact, I appreciate the efforts of the Shadowserver Foundation in depriving Avalanche malware families of their command and control infrastructure by sinkholing the domain names generated by the malware domain generation algorithms. I understand false-positives like this can happen. It comes with the territory. It just sucks that the domain name I was using happened to be a false positive. In this case, it ended well because the Shadowserver Foundation (along with Namecheap) acted quickly on my issue and asked NIXI to have the domain transfer undone.
Having said that, I do believe that you make a very good point. Actions like this should be taken with a lot more care. What if it were not a personal blog but a small business? Depending on the nature of the business, an inadvertent domain transfer like this could have affected the business seriously. The Shadowsecurity Foundation did say that they are improving their processes.
I am not very concerned about this particular foundation when I talk about this. But I am concerned about the systemic issue in the domain name management system that allowed a mistake like this to occur. There could be some type of peer review before a domain transfer like this is executed. I don't think there is a general and popular solution for this problem in the near future. I am hoping that the recent work that is going on in consensus protocols based on cryptography might pave the way to a more decentralized network and more decentralized name management that also become popular and mainstream.
However, if your tweet hadn't gotten traction, and if Namecheap hadn't been proactive, you'd likely have never gotten the domain back. I mean, you had the Namecheap CEO on the case! And for a business losing a domain like that, it'd probably be fatal.
I get that many think that Americans are hugely too litigious. But there is the argument that there ought to be compensation for damages.
You say that "[t]he primary loss I suffered was in terms of time". But arguably your time is worth something. Such as your customary billing rate, times three.
Edit: Or just send them an invoice. At perhaps 50% over your customary billing rate, given that it was a rush job.
Wouldn't the legal fees and time spent litigating exceed the winnings?
But maybe some attorney might do it pro bono.
https://en.wikipedia.org/wiki/American_rule_(attorney%27s_fe...
No, really they aren't. There are a lot of lawyers in America, but most never set foot in a courtroom. They mostly just do "important" paperwork and give advice on following rules.
I know a lawyer who last month wanted to sue someone in federal court, only to discover that Joe Random lawyer is not allowed to file lawsuits in federal court; he had no idea that there was such a thing as a federal trial bar and that membership requires significant experience in federal court under the supervision of a member. He spent a week trying to find anyone that would be willing to file his lawsuit but none would. So he hired a law firm to sue in state court - that is much easier I guess. Anyway, the point is that he is a good lawyer with years of experience doing the lawyer thing, but no experience with litigation. That's normal.
That point of view is rather unfortunate. Why does it have to be about damages? GP even ends his comment on a very postitive note about things that would help. Not every mistake needs to be punished. It was a false positive, and it was heartening to see that all the parties acted fast enough. Why not just move on instead of outraging over hypothetical concerns?
EDIT: I have no idea how to quote parent comments here.
What is your source for that?
If someone attacks you, and causes physical or even mental damage, they typically face criminal and/or civil penalties.
That's common, throughout the world.
> Not every mistake needs to be punished.
Maybe so. But the people responsible for the damage could proactively offer compensation. That'd arguably be the honorable and compassionate thing to do.
> It was a false positive, and it was heartening to see that all the parties acted fast enough.
It's clear that they acted recklessly. So it's not just a "false positive". And what's "fast enough"? A few days after a totally implausible attack isn't that fast. It should have been fixed within the day. After at most a few emails.
MTAs should queue undeliverable addresses for more than four days, so you may not miss much unless someone’s DNS resolver had a poor caching strategy or Shadowserver used a long TTL. At least the name still resolved (else the mail would have been dropped immediately). Some mass mail senders may go to less effort to get their messages through.
The fact that people responded so “quickly” helped here. I put “quickly” in quotes as it was quick enough to possibly cause no message to be lost — I’m sure it was super stressful and didn’t feel quick at all!
Glad you got your name back.
Microsoft Exchange used to (hope it doesn't still?) cache MX records until the service restarted. :( There's a long tail of braindead software in the DNS space
https://github.com/susam/susam.in/commit/91405150ff3f44fd094...
I have created a mirror of the blog on GitHub Pages for you. Visit https://susam.github.io/blog/sinkholed/ to read the mirrored blog post.
The internet in general is infrastructure, and protocols that are increasingly controlled by governments and organizations in bed with governments.
Anyways I've since then stopped buying .in and sticked with .com. I'm glad OP got his domain back, but I'm quite sure it wouldn't have been simple (or even possible) if he had chosen some registrar based in India (due to their incompetence and general nature to not contend with higher authorities) instead of Namecheap; especially since Namecheap's .in domain costs more than that of any registrar based in India.
The basic premise is that unwitting hosts are compromised by malware,this malware is talking to a domain and in order to protect the infected users and curtail the further spreading of the malware the domain is sinkholed.
First, a random authority, regardless of legal relevance has no standing to "protect" infected hosts without explicit consent of the owners. If the infected host is causing harm to other internet hosts then it needs to be taken offline by it's network owner (e.g.: isp or datacenter operator that owns the IP AS number).
Second, in case of malware spreading (e.g.: wannacry) and DGA domains: if the domain is not registered, instead of a sinkhole, an administrative restriction on registering that domain should be placed. If the domain is registered, you want the IP infrasructure to be taken down. IP blocks an reputation damage can be very harmful. IP subnet owners are much more responsive and where that is not the case, a null route can be blaced to "sinkhole" the IP -- null routes are advertised using predefined BGP communities, this means it will be unreachable only by networks that accept that community (e.g.: FBI sinkholes an IP, american networks accept the community and block the IP while other countries might not). You have to understand why there are so many malware domains for C2 and why DGAs exist, it is more costly to become in control of an IP address than it is a domain. If you block the IP as soon as C2 is detected on it,the attacker will just change the A record to point to another IP, but they have a much more limited set of IPs and costly IP infrastructure so they'll be running out of them fast. You can use DGA and dynamic domains such as noip.org (MS famously sinkholed them taking down millions of legit hosts!) But you can't as readily come up with IP addresses. I like this approach because the IP owner is always in a position to force remediation of the C2 server or infected host, they can ban the user or work with them to remediate the infection after confirming, they can request removal from the sinkhole. Most malware operators have no more than a handful active C2 IP addresses but from experience, I see them use dozens of domains,sometimes from different malware campaigns pointing to the few few IP addresses in their control.
I am sure this has challenges but it is a cleaner way of doing it and focuses remediation on the C2. If this approach was taken, OP's IP would have been accidentally sinkholed, her webhost would contact her about it, she would show proof that the server has not hosted malicious content and work with them to lift the sinkhole. Meanwhile,the site can be moved to a different host (if it takes too long) and IP address, since the domain is not being sinkholed it would just work. Malware researchers and law enforcement can see if infected hosts continue to communicate with the new IP or if the new IP responds to C2 initial traffic to decide if it should continue to be sinkholed (costing OP hosting money if it was an attacker, it might cost them money and access to compromised hosts).
I run macOS only and as far as I can tell Nymaim is Windows only. Still, I ran an malware scan on my Macbooks and nothing popped up, so I'm pretty sure nothing infected my devices.
Still, I wonder if I ended up hitting the sinkhole, Virgin was somehow notified and this triggered their email? Or maybe it's just a complete coincidence.
Edit: Sure looks like Virgin works with Shadowserver: https://www.ukfast.co.uk/it-security-news/virgin-media-to-in...
I'm unaware of this particular international cooperation arrangement but it's great to see.
Although my server wasn't infected, it had connected to a Shadowserver sinkhole.
While it's good that there are folks who work to sinkhole botnets, the next step of accusing others of being infected based of what the sinkhole sees needs more care. I'm disappointed that, evidently, my expression of these concerns to the German authorities three years ago hasn't lead to a substantial change at the Shadowserver end.
As can be seen from your case (and mine), you can get blamed even if the software at your end of the connection wasn't the botnet software. Considering how a basic premise of the Web is that it's safe to dereference a URL and everyone runs software that does so (Web browsers!), it's a bad idea that Shadowserver doesn't require a narrower indicator of compromise.
There'd be less chance of folks weaponising this system against bystanders by framing them as botnet-infected if the Shadowserver Foundation sinkhole required the other end of the connection to exhibit more specific hallmarks of the botnet software.
Do I go with a @gmail.com or other corporate address? Then I risk losing my email if my account is suspended.
Do I go with a domain I own? Then I risk losing it if something like this happens.
Either way is serious because email is effectively a master key into all my accounts.
I'm honestly not sure what's best.
I think that's because I was coming at this from the perspective of trying to prevent getting hacked, but really I'm less worried about that than I am about losing access.
This one, it’s this one.
Losing your domain tends to require human action: from someone forgot to pay the renewal to someone messed up and sinkholed it because they thought it was a C2 server.
But because humans are in the system there tend to be layers of processes that try to prevent you from getting to this state and can get your world back to normal if you do.
Gmail offers nothing like this. When the ML algorithms decide you are too many sigma in the “abnormal” category, you are done. And there is no one to talk to who can fix your problem.
If it ever becomes a problem I flip my MX record elsewhere.
Maintaining a domain you control, with a couple of privacy-aware (Protonmail, Fastmail) fallback / out-of-band email contacts, and possibly a few online presences (Twitter, Mastodon, Reddit, etc.) which you can use for signalling, helps.
Knowing actual postal or phone contact information for key partners is also useful. You can use these to communicate in an emergency as well, and spread word.
Operating as a pseudonymous nym online for a decade and experiencing several lockouts / shutdowns over the period ... has been interesting.
• What led to the false positive.
• What actions were taken to notify the domain owner about the actions being taken against them.
• Why there was not a comment put into the Whois entry — or in some other obvious place — saying what had been done to the domain.
I have heard bad things about shadowserver in the past. Now I wonder how much other collateral damage they have done over the years.
Evidently law enforcement was involved in some capacity.
1. shadowserver had an oopsie (i.e. added a false entry to their list)
2. that list was taken by the prosecutor's office to without further checks and by the authority of the prosecutor's office (i.e. state attorneys) of Niedersachsen in Germany and contacted the registrar in India.
3. The registrar transferred the domain
Between each of these steps, there are so many things that went wrong, on so many levels.
Between 1 and 2: Why is the word of some rando org taken as gospel? Without being independently investigated by the office, which sole raison d'être is (independent) investigation of the claims brought to it.
Between 2 and 3: A German prosecutor's office has no authority outside the EU; and even if it did, it could not tell anyone anything to take enforcing actions without a proper court order, even if it's an international request.
Also in the grand scheme of things, considering this was all done to "take down a botnet", this is a path to hell paved with good intentions. How about creating incentive that software and hardware manufacturers get their shit together and secure their shit?
This can happen in any scenario where a special shortcut has been added to get around a standard process (where standard process usually involves some human review and judgement).
I imagine that in most cases, the shortcuts were created simply to speed up a process where some (perceived) harm is significant, and a rapid change would alleviate this harm. This would allow some enforcement agency to rapidly take down a child porn site, for example.
Such shortcuts can also be designed to prevent any other party from knowing what is going on - why the change is being made. This would be the case where some high level governmental agency (FBI, for example) wants to change something to prevent a situation they deem as bad or to perhaps to enable better awareness of a process/communication by inserting themselves into the middle of it.
And finally, in this case, an automated or human error resulted in this person's domain name being included on a "bad list", and the shortcut swept their domain along with the trash.
This particular situation doesn't bother me as much as the "Google/Apple/FB suddenly closed all my accounts" scenario (which is often triggered by some opaque artificial wishful-intelligence system).
Regardless of the situation, it's not ideal that our best course of action in recovering something wrongfully taken is by complaining on public forums. It's a shame that we have to hope for the attention and generosity of someone with more power to champion our case to right the wrongs.
So I say, bring the humans back into the process! :)
This situation bothers me so much more. Google/Apple/FB are private corporations. If they continue to adopt user hostile practices it is possible for some other company to out compete them in the more or less free market of internet services.
TLDs are government controlled entities administering scare resources. I certainly _feel_ like I have much more ownership over my domain name than I do my Google or Facebook account.
This is the history of society, of marginalization of the disabled or other outgroups. It's hard work to overcome that, often through force (Civil War) or the threat thereof (Americans with Disabilities Act).
One of the most terrifying thoughts is that of losing domains to someone who points them at alternative mail servers thus gaining access to everything that uses email for verification.
Also it was very bad on the transparency front. If they are taking down a domain, the operation is not secret anymore, so they can tell why. No telling you is bullshit.
That being a German operation, I would expect much better on the democratic handling of it. And it being an international operation, India should have complained that it was badly done too. What would happen if namecheap didn't care?
By analogy: Google has necessarily-impersonal relationships with millions of Gmail users; but rather more personal relationships with far fewer GSuite and Google Cloud organization owners. If you were an employee of a company that uses either of those, and your service was breaking, you’d ask your GSuite organization-owner (i.e. the person Google has a personal relationship with) to contact them for you.
But when it comes to not telling someone, that's usually because there's some high level legal/law enforcement situation. In those cases, there is sometimes a reasonable argument for not alerting someone. The problem is that if there's no human oversight, innocent people sometimes get caught in the net.
In many cases, sites have resorted to distributing their IP addresses.
“Shadowserver said it was bad”
“But it’s not bad”
“Take it up with them. I hope you speak German.”
They control various seized assets seemingly unrelated to their publicly stated mission, like libertyreserve.com. Shadowserver used to host the seizure page for liberty reserve too.
The investigators would likely argue that notifying domain holders would reduce the chance that they can take down a botnet's infrastructure successfully, which seems likely.
Could there be some maximum time after which the 'rule set' for the auto-takedown code needs to be made open source / public? It must presumably be implemented as software and/or configuration files.
That would at least allow for inspection, confirmation and disputes about how it's implemented, and if this was 30 days or so, it shouldn't risk the takedown effort.
While top-tier network engineers are developing takedowns like this, presumably they'll do a good job of minimizing false positives - but as this case shows, it's not always going to be perfect - and I worry that if it becomes more common, we'll see sloppier implementations.
That could lead to connectivity and access issues for more users (again in an international context). It's great that the situation was resolved in this case but I imagine not all users would be able to raise a complaint at a similar level of technical detail and respectful tone and for it to receive the same amount of attention.
Maybe that's untrue - maybe injustices really do get amplified by social media and relying on companies to notice this 'works'. It doesn't sit particularly well with me as a remediation process though, and I'm not sure it scales.
Is your domain just gone at that point?
Is it really that easy to lose a domain name...by someone doing an 'oopsie'?
I'm not sure that's the correct conclusion to come to from this experience. Yes, the registrant happened to get lucky in this case, in that they had significant enough reach on Twitter and HN to get the right people to pay attention and get eyes on resolving the issue. But that easily could not have been the case (and might still be the case in the future), and with a ccTLD, you have no recourse.
I think the correct lesson here is to go with a gTLD, because if worse comes to worst you will always have recourse through ICANN if necessary (since the gTLD operator is contracted with them). On a ccTLD it's not always gonna work out. Heck, the registrant was already ignored by the ccTLD operator in this case anyway; it's frankly kind of lucky that they had the CEO of their well-known registrar go to bat for them. That's not the kind of intervention you should be regularly relying on to keep and maintain your domain name!
Sure, if you don't pay attention to the care of your domain, it can break in incomprehensible ways, just as if you don't pay attention to the care of your car, it can break in incomprehensible ways.
Do you guys know this stuff? If my domain started resolving to a new IP address, that would be just as unfamiliar to me, as the current address.
Should I ping my domain and write the results down?
But generally, the better your setup is documented, the better you can detect or diagnose when something goes awry.
It can be a good tradeoff to know where/how to find information rather than actually knowing the information.
For a larger one, the netblock(s), ASNs, or hosting. provider / region.
What options are there to prevent this? Would a registrar such as MarkMonitor provide at least some notice or protection?
Really wish they'd pick another name: it's really hard to convince clients to take them seriously and let me choose them. If I had a dollar for every time someone insisted on GoDaddy... :-(
In the physical space, in the states you're free to walk out into the public park, put on a hat saying something atrocious like "I hate cats!" and peacefully petition your fellow citizens to destroy all cats or something silly.
When we moved to printed distribution, there was still a clear bit of guidance; as long as you weren't directing people towards violence, you were good. Most newspapers were locally owned and would even be happy to print your letter to the editor about cats.
But now? We've got a foundation doing something like a regex match on domain names, we've got a criminal element hijacking computers, we've got various government-condoned organizations for managing tlds, we've got registrars. All of these are different types of organizations working in different countries and established for vastly different reasons.
I am reminded of two things. First, Thomas Paine made the point that it was better to live under a dictator than a complex system that hurt you. Under a dictator, you had a guy to point to when things went wrong. It was them! They are responsible for this awful thing! Under a complex system? There's nobody. Bad things just happen, and when you try to ask about it, each party can explain to you that they were working for good reasons to the best of their ability. There was a problem, but no reasonable way to discuss, diagnose, or propose fixes to it.
The second thing was a story from the 80s about a U.S. official, Raymond Donovan. He served fairly well in public office but was accused of some serious crimes. He was destroyed in the media. Then they found out he was innocent. He asked a famous question "Which office do I go to to get my reputation back?"
I'm extremely happy this was resolved, but good grief, if I didn't know anything about the net, and my domain had just been set up instead of being in my control for 12 years, which office would I go to to get my domain back?
Either we own things or we don't. If every bit of our participation online is owned by somebody else, this should be a lot bigger deal than it is currently.
It's easy to say "the dictator is bad" but historically getting rid of dictators without a specific and concrete plan for what to do afterwards has not turned out so well.
In political-theory-land, you assume all actors are bad, are at least will become bad at some point in the future. The question then becomes *how can we organize our political structures so that if we can't prevent dying, can we at least provide guidance to the next government that follows?"
It naturally follows that the real goal is not solving problems: it is providing problems and solutions that people can accept. If nothing else, providing problems and solutions in terms that future generations can reason about.
Interesting; can you point to the quote/source on that? (I'm also reminded of the Jobs quote: "Conspiracy is optimistic! You can shoot the bastards!")
I'm not sure one is "better", per se, but problems with complex distributed systems tend to be painfully intractable. Worse, I think human nature is to map such problems to narrowly-scoped villain narratives which feel tractable; and so not only does the root complexity go unaddressed, but great efforts are undertaken to thwart the alleged villains, sometimes making the real problems worse.
> He explained in his email that my domain name was sinkholed accidentally as part of their Avalanche operation.
Would it hurt the people doing domain takeovers to at least try sending an email to the registrant with contact info and a case number?
P2P systems do not get this by design. For load bearing infrastructure like DNS, escape hatches to stop bad actors putting humanity’s hive mind at risk is a handy tool that has been used effectively.
There are the philosophical questions of “how do we define a bad actor” and “who gets to decide what’s acting in bad faith?” But many botnets don’t suffer from that ambiguity, a well deployed/utilized botnet can take down load bearing internet infrastructure.
Being able to stop future threats without requiring migrating a P2P network _seems_ like a feature, and reminds me a bit of common law. In law, we know there will be cases we can’t anticipate upfront and leave the courts room to interpret. P2P seems to still be in a state where we have to identify and protect against every form of “bad actor” upfront in our game theory in order for our network to be stable moving forward.
[1]: https://medium.com/thisiscala/the-duct-tape-holding-the-inte...
A legal request originating from Germany would have been approved by CERT-IN and NIXI would have had to comply.
https://www.cyberswachhtakendra.gov.in/alerts/avalanche.html
However NIXI is subject to CERT-IN authority which is clearly part of the anti-botnet collective.
https://github.com/susam/susam.in/commit/91405150ff3f44fd094...
This is the fundamental problem with this sort of crowdsourced censorship - there's no accountability on the reporters and no repercussions for bad faith.
Malware researchers reversed a sample of the malware and started blackholing domains that matched the pattern to get ahead of the malware by preventing it from communicating with the domain du jour.
It just so happens that the authors domain pattern-matched domains that would be contacted by the malware.
As the original operation was run by interpol, europol etc i think this was a fairly accurate description, especially in context of a tldr; which gives some constitutional rights to simplify things :)