DataTables CDN Outage – post incident review | Hacker News Reader

DataTables CDN Outage – post incident review

datatables.net

42 pointscristoperb27 comments

Didn't expect to see this here, it was over a month ago this incident happened! Happy to answer any questions about it (author of DataTables here). It was a super stressful event to say the least, and I've been reading along with the recent npm incidents wondering what I can do to make sure my OpSec is as good as it reasonably can be.

Maybe because your Blog RSS [1] shows releases only, it doesn't seem to show these interesting tidbits?

[1] <link rel="alternate" type="application/rss+xml" title="RSS 2.0" href="http://datatables.net/feeds/releases.xml">

The blog feed is here: https://datatables.net/feeds/blog.xml . It is advertised on the landing page, but it looks like I've missed having it on the blog page! As you say, that has the releases feed - thanks for pointing that out.

It would be helpful if you would share the name of the registrar so that other people could be aware that this policy exists if you work with that registrar.

Joker.com. Credit to them they fixed it reasonably quickly, but its a horrible policy to default to enact the change if no response if given. Their reasoning was what else would they do if someone got locked out of their email - they need a way to recover their domain somehow, and they ask for ID to be submitted, but as seen, that is trivial to fake.

The only real solution is to tie the accounts to the digital identity of a person/company and enforce strong authentication for these cases. Not sure if there's already some EU level solution to this. This is of course pretty complicated to implement, but it would be a valuable extra service for customers.

Totally OT, but thanks so much for DataTables! I used it for a tiny personal project a few years back and it's been quietly chugging away with barely any maintenance required. It was so easy to get up and running with the documentation, implement and customise to my heart's content -- truly an excellent piece of open source!

That's awesome to hear - thank you :-).

Hi Allan, I am a big fan of yours, found datatables while working on personal project of mine last year. Its perfect.

Using it in another personal project this year.

Oh, hey! I discovered your library around a month ago, and had a question at the time [0]: why is it mostly sponsored [1] by personal injury lawyers? Are they particularly heavy DataTables users? Or is this an SEO thing for them, since the top sponsorship package comes with a site link?

[0] https://bsky.app/profile/spinda.net/post/3lx3xkzbc622t

[1] https://datatables.net/supporters/

An SEO thing for them. Useful income for me. It isn't much ($49/year), but every little helps...

Hell Allan, apologies for going fanboy on you but just wanted to tell you that DataTables is amazing and we use it a lot in my circle of friends. You made an awesome product!

Out of curiosity, could this have been a vector for a supply chain attack?

I am currently running an fairly outdated version of datatables on a personal project, v1.11.3 from 2021. I'm not too worried about running this older version, because according to dependency scanning software there's no CVEs for it [1]. Also, upgrading this package is too tricky as there's been some pretty huge breaking changes, so I'm stuck at this older version.

I am _not_ using the datatables CDN but instead self-hosting the static files. However, I did not notice until recently that in v1.11.3 it comes with a CSS stylesheet [2] that loads a static resource from that CDN: `url("https://www.datatables.net/examples/resources/details_open.p...")`

It looks like newer versions of datatables don't import static files from the datatables CDN like this.

Presumably if this domain was hijacked as stated in this incident review, users on affect datatables version could have had their site compromised?

Would it make sense to issue a CVE for older datatables library versions that could be susceptible to this attack?

[1] https://security.snyk.io/package/npm/datatables.net/1.11.3

[2] https://cdn.datatables.net/1.11.3/css/jquery.dataTables.css

> Out of curiosity, could this have been a vector for a supply chain attack?

If you were using the CDN without SRIs, then yes, that would have been the most obvious channel. However, I don't believe the attacker ever set up for that and the URLs never resolved due to CloudFlare blocking it.

> there's been some pretty huge breaking changes

Unless you were using the legacy API, there shouldn't be any major impediment [1]. I intentionally tried to keep backwards compatibility as I hate doing library upgrades myself! Drop me an email - allan at the domain in question if you have any questions about doing an upgrade.

> It looks like newer versions of datatables don't import static files from the datatables CDN like this.

I rewrote aspects to use CSS styled elements in place of images, so there were less resources to load.

> Would it make sense to issue a CVE for older datatables library versions that could be susceptible to this attack?

Per the above, if you were using the CDN without SRI for the resources, then any version could have been susceptible. However, I've seen no evidence that the attack took that vector.

[1] https://datatables.net/upgrade/2

I seem to recall enjoying using datatables. You, or somebody else associated helped me on the forums. Not sure what I asked but I remember two things: positive dev interaction, and the pain of figuring out how to make the OOX/Excel export not lose proceeding zeros. (Had to write my own handler to change the xml)

Thanks for the pleasant reply!

I thought I was not using the CDN as I had self-hosted the static sources, but some image sources seemed to be imported from the CDN in stylesheets in the version of data tables I linked.

I just updated my application from v1.11 to v1.13 without any trouble (aside from some minor aesthetic changes to padding), so at the very least I now benefit from your styled elements.

Thanks for your dedication on this package, I’ve used it for years and it works very well.

It’s still a complicated attack and I can understand the registrar being confused, though they should’ve called you for sure.

> They used an email address intentionally crafted to look like it could be mine and submitted a fake driver's license and utility bill with information that could only have been from leaked WHOIS data. The registrar accepted this as proof of identity and started the transfer process. That included sending an email to me to confirm the transfer, an email which I never saw due to the flood of emails (which it is now easy to say was the start of the attack).

Edit: Cloudflare blocking the attackers code with a 1000 error is interesting. Could you share some information about it?

Yeah - it was a well set up attack. What I don't understand is that there was no obvious follow on. I can only guess that it was a proof that it could be done. Maybe?

Regarding the 1000 error - I didn't have any 1:1 support contact with CloudFlare - the first I knew was they were returning 1000 errors, which I presume they were doing due to a blacklisted IP being used for the DNS resolving. I'm really not sure though.

The takeover due to the lack of response to an email is worrying

Yeah, I really wasn't happy about that. I did put it to the registrar that such a policy is wrong and open to such an attack. I got the impression that they weren't going to change their policy though. Such policies are something I'm going to be looking at when considering a new registrar.

Who is the old registrar?

https://who.is/whois/datatables.net

> The takeover due to the lack of response to an email is worrying

The trouble is that that is the way most way modern small print is worded.

Read the small print in any contract of the last 10 years. Almost all of them when speaking about delivery of notices will apply $n days to emails.

Everyone in the tech world knows why it's a horrible idea, but sadly most lawyers who draft these things either don't care, or their client doesn't care and that's how the lawyer has been instructed to draft.

This is perhaps a lesson that people should use the extra domain security functions, e.g. Domain Lock which is available on most (all ?) TLDs.

If your registrar does not expose the functionality, move to one that does.

N.B. Ideally you want Domain Lock == REGISTRY-LOCK, there is also REGISTRAR-LOCK which is similar in concept but not quite as secure because REGISTRAR-LOCK is implemented at Registrar not Registry level.

> The fact that someone would attack an open source product such as DataTables sickens me. I release by far the majority of my work as free open source software, host a free to use CDN, and support the software.

Seriously, no idea what could motivate this, unless a paid datatables vendor felt you were undercutting their business. We all like to think that attacks are beneath them, but stuff like that has happened before.

Maybe the real target was some application that was using this component through the CDN. If you take over the CDN, youn can inject your code into the application and maybe steal something valuable.

That's the only reason I could think of for doing this, but I saw zero evidence that it was done that way. Baffling!

Reply on news.ycombinator.com