Three Days in August: What a DDoS Attack Exposed in Our Network | Hacker News Reader

Three Days in August: What a DDoS Attack Exposed in Our Network

nine.ch

19 pointsnine_ch25 comments

> There was no unauthorised access and no compromised systems. This was an overload attack, not an intrusion.

Hopefully your logging infra is rock solid and nothing has been dropped in the flood. It wouldn't be the first time a DOS was used to mask the actual attack by overwhelming the monitoring infra.

> Use a CNAME or ALIAS record instead of an A record. An A record ties your domain to one specific IP address on our platform. That fixed binding was exactly the problem during the attack: wherever we could change the address on short notice, availability could be restored, wherever we could not, only the blunt measure remained.

I don't understand how this helps. CNAMES have TTLs like A records and they eventually have to terminate at an A record somewhere so why pay for an extra hop?

> I don't understand how this helps. CNAMES have TTLs like A records and they eventually have to terminate at an A record somewhere so why pay for an extra hop?

I assume the customer controls the domain DNS records here rather than the hosting provider.

CNAME record: hosting provider can change underlying IP freely.

A record: hosting provider must get in touch with customer to change DNS.

I probably incorrectly read

> we provide that customer’s connection to the internet

as providing the DNS infra for their customers.

indeed, this (A vs CNAME) is nonsense. I applaud this provider's transparency but they missed a 4th gap: inadequate network security expertise. Two of the 3 gaps identified are perhaps not baseline but they are well understood hygiene. They shouldn't have had to learn these things as a result of an incident. (the 1st gap, not monitoring customer network blocks, is very understandable and i find no fault there.)

and then up the stack a bit, they are confused about DNS' role in attack mitigation, at least generally speaking. perhaps there is something specific to their own DNS setup where this made a difference. they should just strike that part of the PM entirely.

that said, their reaction time is amazing. this would have included live troubleshooting during an ongoing incident! criticism aside, i wouldn't hesitate to use them if I wanted EU service.

> There was no unauthorised access and no compromised systems. This was an overload attack, not an intrusion.

"AI said it's all good. There are no attackers within our walls."

I sometimes work on a system so byzantine and poorly-documented with so many horribly obsolete systems - they have Windows XP machines all over the place - that there's no way it hasn't been popped.

I think it's safe though.

I think various groups have popped it, taken a look around with growing expressions of horror on their faces, and gotten back out as quickly as they could. Nah man, no way, not touching that, don't want my prints on this particular gun, there's no way this is legitimately how they've left it, this has to be some kind of a setup.

Security by obsolescence layered with security by complexity.

it's not enough to salt your hash; you must mold your Swiss Cheese

It's encrypted using SHA-255; in theory half as secure as SHA-256 but in practice they spend ten times long trying to work out why nothing is working properly.

you can fit 255 into a byte tho, giving you room to do more encryption

Being semiserious I do wonder how hardcore you could do encryption in 8-bit processors. Like, how hard would it be to keep "nation state" level cracking out of your private messages for a week or so?

You could probably generate elliptic curve keys pretty effectively even with an Apple II, at least enough that you'd have to throw a couple of dozen Israels worth of intelligence agency at it to pop it.

Enterprise security in depth.

"We found three concrete gaps during this incident, and we would rather be upfront about them than gloss over them." claudism?

Sticks out like a sore thumb, doesn't it?

I would have never suspected that one tbh.

Damn

How do people even write anything anymore? I would be so afraid of writing something that someone would suggest I wrote with AI. Reason being, I guess, is that I'm just so surrounded with it and US corporate speak that I rarely notice it anymore.

I read so many Claude walls-o-text I find my own writing ends up using the same tropes. And so, I end up "using AI" even when I don't.

is it? i would write a sentence like this on my own. i guess informed by the thousands (or hundreds, at least) of PMs I have read or contributed to. the same material LLMs would have trained on.

Naive question: services that can be used for amplification attacks, are they constantly getting patched to prevent the latest iteration of attack type?

In other words, if there are a bunch of services prone to amplification attacks, can traffic from these services be upstream-blackholed for the duration of the attack?

If it's not traffic coming directly from an IoT botnet, which is probably where the source of the spoofed traffic that initiates the amplification, then isn't there likely a smaller, more manageable number of services responsible for the attack traffic?

Or are we talking services that form the substrate of the internet that have inherently exploitable protocols that it would take a large herd of organized cats in order to update in a way that doesn't break the internet, and will still take ~10 years?

I still think in IPv4, so this may be a stupid question, but it's it known how many unique IP addresses were attempting to connect in the space of that time, and then it's there logging to identify those with unusually large amounts of individual traffic?

Ten years ago, when I worked on stuff that attracted DDoS, the memorable vectors were UDP chargen reflection and wordpress pingback reflection.

Wordpress does get lots of patches, but I don't know how you really fix pingback, but it was easy enough to look for user-agent WordPress and drop requests before serving large files (or really anything... what do I have that WordPress should request). For a smaller site, the volume might have been high enough to overwhelm TLS handshaking, which is harder to solve.

Chargen, wow. There's pretty much zero need for it to be on the internet. There's no need for anyone to run it. But evidence showed many instances running and it seemed to be the implementation Microsoft shipped in the Services for Unix package. Someone was trying to blocking the reflected traffic, but the servers were sending 64k responses (!) and that was being fragmented, and they only dropped the first fragment... fun times.

I didn't spend time trying to get the hosts involved to stop sending this garbage... Writing abuse reports is herding cats, and networks that would be responsive probably already have taken these senders offline. I was also seeing short duration attacks consistent with people trying the free tier of DDoS as a service... so dealing with 90 seconds of garbage every once in a while was no big deal (as long as fragment reassembly didn't knock the machine over)

> services that can be used for amplification attacks, are they constantly getting patched to prevent the latest iteration of attack type?

unfortunately, no.

Did AI write this?

Yes. The blog version was drafted with AI assistance and then edited and reviewed by us. The facts, numbers and timeline are ours and match the technical postmortem PDF linked in the post, which is the drier, more complete version. Happy to go into any of the technical details here.

  8. 80.239.216.210.  0.0%    78  123.2  64.6  42.9 141.8  30.6
  9. vl202.zur-itx1-dist-1.cdn77.com. 0.0%    78   60.1  58.8  43.7 136.5  24.2
  10. 89-187-165-194.bunnyinfra.net. 0.0%    78   45.9  63.5  41.8 131.9  31.0

so cdn77.com and bunny.net

Yes, that's bunny.net, it's named in the post. Their Zurich PoP appears to sit on CDN77's network, which is on their side of the setup, not something we configured.

We're a Swiss managed hosting provider and run our own network. In August one of our customers was hit by a UDP amplification attack that we estimate peaked at 500 to 600 Gbit/s across all our links combined, several times what our uplinks can carry. Because we provide that customer's internet connection, the impact hit our network directly, and roughly three hours later the attack was broadened to our own services as well.

The write-up is mostly about what we got wrong: our automated detection only covered our own prefixes, not customer prefixes we announce on their behalf. Blackholing at the IXs we connect to only took effect via route servers, not on direct peerings. And we had no way to withhold a prefix from one specific upstream, so we had to build that while under full load. On top of that, our own website runs on the same shared platform as customer apps, so when it was targeted, unrelated applications were affected too.

What eventually ended it was moving exposed applications behind a CDN with DDoS protection. The full timeline is in the PDF postmortem linked from the post. Happy to answer questions.

Reply on news.ycombinator.com