The Hijacking of Perl.com | Hacker News Reader

The Hijacking of Perl.com

perl.com

298 pointsleejo107 comments

I had a very hard time reading this article. It's filled with so many platitudes and truisms ("And, it always helps to have friends and good relationships with the people who are able to help.") and yet it doesn't really explain what happened. For all I know they might have forgotten to pay for the invoice.

Yeah it's not very good writing, but the tl;dr is that someone executed a social engineering attack against Network Solutions, got them to update contact information for the domain owner to themselves, transferred it to BizCN, then transferred it to Key Solutions, and then attempted to auction the domain name for $190k.

It reads like the author is basically guessing, in the lessons learned he mentions 2FA and in the same sentence he mentions that it might not have helped ultimately. He mentions it's a speculation, he'c not the injured party. Someone else did some "forensic work" yet the link doesn't work. I feel this should have been written by the domain owner or the registar.

Couple of other issues: "It’s important to have one face (mouth?) to represent the diligent work everyone was doing." From the registar article: "the Perl team has yet to respond to our request for a comment"

I don't know who The Register tried to contact, but it wasn't me. As soon as their article came out, however, I contacted the reporter.

And, I've fixed the broken image. Thanks for noticing that.

> It reads like the author is basically guessing, in the lessons learned he mentions 2FA and in the same sentence he mentions that it might not have helped ultimately.

If the attack was mounted via faked documents which is what I think the article implied then 2FA would not have helped as it was attack on a procedure of replacing contact info.

grep "What we think happened"

I agree the article did a bad job keeping me interested, but I think it is more of a statement about the rumor mills and fake news about the takeover and to be cautious about falling into it.

At the very end they speculate it was a social engineering attack at their registrar that started last year and affected multiple domains. Oddly, the author says they weren’t ”the injured party” and therefore didn’t ask the registrar to verify that. I’m not stating the registrar because there’s no supporting evidence that they were at fault but the author gave it in the article.

He says he himself was not the injured party, a legal term, so the information was not provided to him and he does not have a legal right to it.

This. There's only so much he can talk about from his side, and whoever legally owns the domain will have to provide information on what exactly happened-- should he wish to. That's also important, the owner may not want to talk about it in public (but that would just make the initial blog post even stranger, so I'm assuming they eventually WILL speak up about the rest..)

Sorry but individualization and soul and style and wit are American ideals. If you want drab Soviet style information only, look elsewhere.

I thought this article was interesting from an incidence response point of view, perhaps it have been better framed with a different title.

I mean, the second paragraph starts with "First, this wasn’t an issue of not renewing the domain."

The article was changed since my original comment.

Aaahh dang, I see!

OK, at least it isn’t only me.

Wow I haven't heard the name Tom Christiansen in years. I remember when he used to comment on Slashdot. This inspired me to find my slashdot login and dig up some of his old posts. I just discovered his amazing eulogy for Gary Gygax [0]. I never knew that Tom used to work at TSR (of D&D fame) before becoming a programmer.

[0] https://slashdot.org/comments.pl?sid=475216&cid=22665150

He’s still active online, but his online activity currently seems to be concentrated at the Stack Exchange site on English Language & Usage:

https://stackexchange.com/users/216196/tchrist?tab=activity

I often end up on his answers on StackOverflow when searching things related to Unicode. [1]

They tend to be complicated "nothing's easy about Unicode" type answers, which some users call out as pedantic. But I much prefer getting the full picture from an expert like him and then making pragmatic compromises myself where needed, instead of the usual quick and easy answers that end up being full of hidden traps.

[1] https://stackoverflow.com/users/471272/tchrist?tab=answers

I too love it when people go overboard on a stackoverflow question and get pedantic. Usually someone else gives a tldr; but I go down the rabbit hole and try to learn something ina addition to the short answer along the way.

He was one of the big names in the Perl world back in the 2000s. Wrote some books (including the wonderful Perl Cookbook) and I remember having a course at work given by the Tom Chritstiansen Perl consultancy.

A lot of domain name management runs on honour system. Here are two relevant stories regarding this:

- The Duct Tape Holding the Internet Together: https://medium.com/thisiscala/the-duct-tape-holding-the-inte...

- Sinkholed: https://susam.in/blog/sinkholed/

Disclosure: I am the author of the second story.

Interesting read. What was your conclusion to the below?:

I also wondered if a domain name under a country code top-level domain (ccTLD) like .in is more susceptible to this kind of sinkholing than a domain name under a generic top-level domain (gTLD) like .com.

That appears to be answered in the following sentences:

> I asked Benedict if it is worth migrating my website from .in to .com. He replied that in his personal opinion, NIXI runs an excellent, clean registry, and are very responsive in resolving issues when they arise. He also added that domain generation algorithms (DGAs) of malware are equally, and possibly more, problematic for .com domains. He advised against migrating my website.

I am still using the .in domain name since the sinkhole issue was resolved in Dec 2019. I haven't faced any issue again. So that was a total of 1 issue in 14 years of using a .in domain name. I am not certain if I can draw any conclusion yet other than what is already mentioned in my blog post which @vxNsr has quoted in a sibling comment.

If you own a high value domain, you should consider asking your registrar/registry to turn on a registry lock [1] which protects you from compromises or social engineering at your domain registrar. It's a little more expensive and can slow down NS delegation updates, but otherwise you run the risk of what happened here to perl.com, which can be extremely disruptive even if your attackers don't try and resell the domain.

You can check the status of a domain by looking for "Status: server{Delete,Transfer,Update}Prohibited" in the whois response for that domain [2].

[1] https://krebsonsecurity.com/2020/01/does-your-domain-have-a-...

[2] https://www.verisign.com/en_US/channel-resources/domain-regi...

I know NetSol in theory supports registry lock, but last time I checked they want >$1000/year for it, and it's kind of shitty they don't offer robust access controls internally so you end up paying for it (and other registrars offer registry lock (and hopefully competent 2FA on top of that!) in the ~$500/year range)

Pairdomains (pairdomains.com) offers it for $0.00/year.

But…be absolutely, 100%, certain that the information contained in the registry record is 100% accurate for name of registering organization and contact information. Because the process to unlock can be quite…difficult if the information is slightly off.

An unfortunate side effect of defending against social engineering attacks that can use very small inaccuracies to be successful.

pairdomains.com doesn't have serverUpdateProhibited, which is the "registry lock" protection. The reason why it costs money is because I believe it involves the registrant, registrar and registry coordinating a manual unlock out of band, so in theory if the registrar-registry API is compromised, you're still be protected.

There's zero reason to still use NetSol in 2021 except sheer masochism. Namecheap offers 2FA and registry lock for free.

Seriously, transfer your domains to almost literally any other registrar. They'll be better than NetSol.

I believe Namecheap only offers registrar lock (clientUpdateProhibited), not registry (serverUpdateProhibited). There are very few registrars that offer registry lock and they're all "enterprise", probably because the relationship required for registry unlock protocols to work doesn't scale with retail customers.

Off the top of my head: NetSol, MarkMonitor, CSC, maybe Cloudflare. There are more that will do it for specific ccTLDS (.ca has lots, for example).

Huh, looks like you're right about serverUpdateProhibited. TIL! Thanks.

We nearly had something like this happen to Fastmail many years ago:

https://fastmail.blog/2014/04/10/when-two-factor-authenticat...

Scary stuff. Basically we had 24 hours to dispute via email when a fax was sent to our registrar with a faked up Australian company registration and a fake passport asking to remove 2FA and change the owner email to an address @qq.com.

At the same time, our hostmaster email address had been signed up to hundreds of non-double-opt-in mailing lists, so that there was lots of noise for this email to be lost in.

We had to fight very hard to be allowed to see the fax that was allegedly from us, so that we could see what they had done.

Aside from the primary content regarding the hijacking the registrar, I really enjoyed reading about the methodological approach they adopted for tracking information and contacts during the crisis.

But to the primary content - I've been surprised at just how ad-hoc much of the internet backbone infrastructure is as I've learned more about it. The same could be said about the payments processing industry! Beneath all the complexity and sleekness underlying the tools we use every day seems to eventually lie a system of IOUs, with an honor-based resolution mechanism between sufficiently trustworthy entities.

> Beneath all the complexity and sleekness underlying the tools we use every day seems to eventually lie a system of IOUs, with an honor-based resolution mechanism between sufficiently trustworthy entities.

This is how societies - in the end - work. It is all about trust, I think.

And although there are forces to undermine that fundamental trust, it does still work.

There really isn't a different way to work honestly. Either we rely on trust or force and that trust and force can be transferred and dissipated through a legal system but force loses its transferable value very quickly while trust can be passed on many times.

It's how the fiat money system works as well.

If you recall from January, even with the US election you'd only need a conspiracy of a few hundred people (congressmen) to completely overturn the election for whatever result you want.

Society is made out of string at best

That conspiracy was surprisingly widespread and the normal societal blocks didn't manage to engage early enough in the process to stop it but, in the end, we made through it alright due to societal pressures at large.

I don't know what specifically would've tipped it the other way but I think there were several large players able to extend effort that ended up not doing so simply due to a lack of need (i.e. the military in a non-show manner).

I believe that the point was that if that group decided to overturn the results, the democratic will of the entire population would not have overruled them. The intent of those provisions is for a case like 1877 when the democratic will of the people was actually in question.

Unanticipated in the Constitution was that a group of violent people would attempt to force Congress and the Senate to overturn the results. Had the protestors succeeded in that goal, it would have effectively ended the Republic. We would continue to be a Republic in name, just as Rome was after Julius Caesar crossed the Rubicon in 49 BC. But not in reality. And not too many years would pass before even that charade ended. As happened in Rome when Augustus Caesar became the first real emperor.

There's also a grassroots problem: the reason it took desperate acts in the endgame to try to overturn the election was that election officials around the country refused to bow to pressure to change the results at the local and state level.

That layer of defense can't be taken for granted, especially given how aggressively hostile towards reality itself many local GOP officials are becoming.

> John Berryhill provided some forensic work in Twitter that showed the compromise actually happened in September. The domain was transferred to the BizCN registrar in December, but the nameservers were not changed.

Isn't this preventable with "clientTransferProhibited"[1]?

> This status indicates that it is not possible to transfer the domain name registration, which will help prevent unauthorized transfers resulting from hijacking and/or fraud. If you do want to transfer your domain, you must first contact your registrar and request that they remove this status code.

If nothing else, you'd think that some simple monitoring would be warranted if you own an important domain, like checking the exit code of:

# whois -h whois.verisign-grs.com google.com | grep "Registrar: MarkMonitor, Inc."

[1]: https://www.icann.org/resources/pages/epp-status-codes-2014-...

If the domain thief has access to the network solutions account, they can just remove the domain lock. They have the keys to everything.

There are more secure registrars than network solutions that require much more to transfer, like others have executive lock. You can specify certain terms that must be done before a change. Like the registrar must call you on a certain phone number and get a password verbally.

At DnProtect, we are aware of at least 20 domains that have been stolen since the beginning of the year. Most from network solutions.

I don't understand the expected outcome from this attack.

The hijacking of perl.com was front page news for the technical community. Did the thief really think they'd just be able to drop it on Sedo or Afternic and be done with it?

Ransom maybe?

Nah, ransom is not usually the goal by domain thieves. They steal them to make a quick buck by reselling them.

I don’t think it’s really an attack on the domain Perl.com. Rather, Perl was just one of the domains stolen at the same time as others.

I’m aware of at least half a dozen or so domains that were stolen at the same time, by the same domain thief.

This was not an attack or hijacking. It was the stealing of domain names.

What these domain thieves do is steal the domain, transfer it to another registrar, then attempt to sell them.

In this case, Perl.com just got caught up in a list of domains that were stolen at the same time. Others stolen at the same time also start with the letter P.

> We think that there was a social engineering attack on Network Solutions, including phony documents and so on. There’s no reason for Network Solutions to reveal anything to me (again, I’m not the injured party), but I did talk to other domain owners involved and this is the basic scheme they reported.

Look, if your domain is with Network Solutions, and you missed the other wakeup call[1] to get off of them; let this be the wakeup call.

Network Solutions was the right (only) choice for domains in the 90s, but it hasn't been the right choice for domains in probably two decades.

[1] https://www.theguardian.com/technology/2013/oct/08/whatsapp-...

Key-Systems GmbH seems like a legitimate business to me (judging by their website and company register data), seems they acquired the domain from the Chinese registrar to resell it. Still, seems hard to believe that you wouldn't become suspicious when a Chinese company offers you a very popular domain name that seems to be in active use for sale.

That said I've seen registrars make some glaring mistakes in the past and many still rely on faxed documents to authorize domain transfers, so it's not a surprise that stuff like this happens. Often, all it takes is finding out who's the registrar (easy), obtaining a blank transfer authorization form from that registrar (easy again), obtaining the personal or company data of the domain owner (a bit more difficult but still doable), fill out the form and fax it in. Some providers won't even bother to send you a notification when transferring the domain, so like here the legitimate owner won't notice it's gone before it's way too late.

Key-Systems is a huge registrar. It is very unlikely that they acquired this domain themselves, more likely it was a client of one of their many resellers.

Neither of the registrars “acquired” or “offered” anything. They simply accepted transfers from some fraudulent registrant, and there’s absolutely no reason they shouldn’t allow transfers of domains “in active use”, popular or not. It’s just business as usual for them. The domain was eventually listed by the registrant on Afternic, a domain marketplace. Again, neither of the bounced-through registrars got anything to do with the listing and reselling.

This highlights a usefulness of not choosing the largest and/or cheapest domain name registrar. I work at a small registrar, and we know all our customers and communicate with them directly. Social engineering attacks get harder in such an environment.

Which registrar do you work for?

(S)He can't say or they risk growing too big ;)

I have been asked this before here on HN; my answer is here:

https://news.ycombinator.com/item?id=21437951

Very sensible answer :)

Hmmm... in that case I wonder if such offerings exist in the US?

> This highlights a usefulness of not choosing the largest and/or cheapest domain name registrar.

Since the domain is that old ("This domain was registered in the early 90s" according to the article), was there really a choice? IIRC, back then the only domain name registrar available was Network Solutions.

That might be the reason for the initial choice, but there's doesn't require staying with the same registrar now.

   Creation Date: 1994-08-16T04:00:00Z

It doesn't take a lot of time to move to a new registrar. And it should be seamless as long as you aren't relying on non-registrar services from the old registrar and you set the same nameserver settings at the new registrar.

So what actually happened? Neither OP nor linked press articles seem to really explain it.

(There's a joke somewhere in here about how readable Perl is.)

Skip down to the section labeled "What we think happened".

Essentially, someone took over the Network Solutions domain management account for the domain and transferred it to a different registrar.

It's not clear if that was due to a weak password, compromised contact email account, someone social engineering the Network Solutions staff, or something else.

What actually happened is that a bunch of domain names were stolen at network solutions. Many of which start with the letter P. Perl was just one of them.

We probably will never know how it was done, as network solutions won’t say, and they shouldn’t.

OK but it was probably social engineering.

The article reads kind of like those online recipes that have 2 pages of "My mom grew up on a farm, and her mom liked buttermilk, so this is the story of how this recipe began..." If all you want is the recipe, you need to scroll past that long intro (which might be interesting, but it's not what you are looking for).

> And, it always helps to have friends and good relationships with the people who are able to help.

It would be nice if, you know, people just did their jobs impartially regardless of whether they know or like you. But the reality is that not knowing the "right people" does indeed make things much harder, as we hear often here on HN from small businesses trying to deal with the tech giants.

Cronyism is embedded in most cultures

Unfortunately, it turns out to be really tough to handle every possible edge case correctly without a little bit of cronyism.

People forget passwords, lose 2FA devices, etc all the time. Many of the usual methods can be hijacked much more easily than you might think. It's tough to make the right call for legitimate user who messed something up versus a particularly sly attacker every time without some out-of-band personal knowledge about the person in question.

All of the fancy tech in the world can't beat "Hey, I know Tom pretty well, this doesn't seem like something he would want to do. Maybe I better ring him up through a medium I trust to confirm this before I do it."

I had this same thing happen at my company, Godaddy somehow allowed someone to disable dual auth through social engineering and reset our password through a compromise email. They proceeded to initiate a domain transfer. Not sure how Godaddy would allow disabling dual auth over the phone.

I’m surprised that the domain name ownership hasn’t been transferred to the Perl Foundation or something. Things might look ugly if Tom suddenly dies.

Why? Perl.com doesn't have anything to do with the Perl Foundation. It's a collection of blog articles from various Perl developers.

The website for Perl is perl.org and was not affected at all by the perl.com hijack.

The Perl.com website is already managed by the Perl Foundation. I feel like it's important enough that it warrants being under the full custody of the foundation.

No, the Perl.com website is managed by David Farrell. The domain is owned by Tom Christainsen. The Perl Foundation is not involved with the operation of either.

The "about" section of the website says

> Since 1997 Perl.com has been the home for quality articles about Perl programming, news and culture. The website is managed by the The Perl Foundation.

They might want to update this if it isn't the case.

I literally shuddered when I read "Perl NOC". It was like the ghost of a neckbearded BOFH breathed down my neck... On a serious note, I absolutely adore the simplicity of their blog (https://log.perl.org/)

A bit off topic, but I like the usage of "social engineering attack" instead of "anything to do with the word computers/cyber/hacking", because it places the onus on the correct parties and the correct systems that failed.

A slow heist by Chinese spammer/scammers to use a popular domain name. This is clever and in direct contrast to the usual gobbling up of domains for spam/scam purposes as soon as they expire at the peril of a forgetful owner.

Why try to assign a nationality to the thieves if that is not known? Domain theft is not very clever (there's a record at every step) and very much reversible. Being the buyer of a stolen domain is a bad spot: you will lose the domain and your money, too.

Perl.com seems like it might be connected to other things, perhaps the sale was just to hide something more nefarious? I don't know how many @perl.com email addresses there are but if people submit code using email as an identifier ... I'd guess there are second factors and such; and that in this case people now know to check for errant submissions/accesses. AFAIK there's nothing juicy to download from the perl.com domain itself?

I did not read this, But a mate I used to work with was getting harassed by debt collectors, He looked up the DNS Records and the domain registrant email was set to another domain. admin@suchandsuch.com, Well turns out suchandsuch.com had expired.

He bought the domain, set up the email admin@suchandsuch.com and reset the debt collectors main domain by requesting a transfer to another domain provider, it then sent an email to the domain record holder and he took over their domain.

Never use the same domain or another domain for registrant emails and buy privacy. use a gmail or your isp provided email.

That is an incredibly long winded way of saying very little.

Which is ironic as that is literally the opposite of the Perl programming language itself.

Nice work, Perl.com team! And, a very helpful incident report we can all learn from.

The mentioned BizCN registrar has a very shady past according to a simple web search.

I don't entirely understand how it was discovered. Was Perl.com pointing to something delivering content obviously not right for a while?

A very long article with no real information. Why did they even decide to publish this weird thing?

Took me a while to figure out what this “Perl” was, so if you’re like me, I’m gonna save you some time: “Perl” seems to be the old name of the popular Raku language.

Perl and Raku are two individual languages. Perl is still very much alive https://www.perl.org/

I think he was making the funny

The use of the word "popular" made me wonder this. Either way, happy to clarify the facts in the case that other people didn't get the joke.

Jesus, this makes me feel old. Perl's been around for decades, being a very common CGI language pre PHP days and also for general sysadmin work.

Perl's pretty much dead, I haven't written any perl all month

Perl is now alive. 100% of the code I've written this month is perl.

I see what you did there!

I wrote in Perl, Python, and Julia this month (yeah, 1st day). Probably will have some C++ and R as well later. We'll see.

I still use it as my primary language, and with CGI too.

It's lightning fast, it's mature, every problem is solved, it still works great out of the box, and compatible with just about every web server out there.

By "mature" I mean that any question I google there is real content, and not just StackOverflow. Real web-based knowledgebases which also happen to be lightweight HTML, without JS required, and with real solutions to the problem.

I like perl, but the temple you build with it becomes very complicated. Sadly the fortress of TCL has won me over.

I try to write it in "PHP style", meaning just a whole bunch of "my $someVariable = SomeFunction()", and as little as possible of "=~ /^#^$^&*&/ <> ;~;"

It helps that I try to build and maintain two versions of the same codebase, so I tend to write in a language which is the lowest common denominator of PHP and Perl.

I'm at a modest 14K lines now, and still find that I can find what I'm looking for with just a global text search...

You made my day, thanks!

Are you being sincere?

What would solve the constant fear of losing important domains is making domains NFTs on the Ethereum blockchain.

This would make the situation better in two ways:

1: A normal domain move can only happen when the domain owner signs the transaction. If the domain owner claims to have lost their key, this would raise a red flag and result in an in-depth analysis which the domain owner has to pay.

2: The movement of the domain would be announced on the block chain. So in case the in-depth analysis has been tricked by an attacker, the righteous owner would be alarmed immediately. For this they would use some service that monitors the blockchain for them. They could then reverse the transaction with their key.

https://app.ens.domains/

No need for some fancy blockchain technology here.

You can't move most domains without written consent of the domain owner and you'll have to properly identify yourself before requesting that.

In the end it's about how the registrar handles those things.

Define "move" here. I move domains between registrars every now and then and I never have written anything on paper to do so.

If I want to move .de domains from one hoster to another, I'll need authentication codes and alike for that. This process has been automated and digitalized but the current domain owner has to explicitly acknowledge and allow the transfer.

This is how DENIC handles those things at least.

No, this is just yet another thing that "the blockchain" does not solve.

I actually can't tell if you're being serious, or if this is satire about the fact that people think "the blockchain" can just be sprinkled and solve any problem.

Absolutely right.

I'd certainly pay for this service.

Imagine domain names actually being secure, like bitcoin is secure.

I've posted this idea here before and also got shat on and downvoted to oblivion.

Reply on news.ycombinator.com