Belgian ISP under 250 Gbps DDoS for days on end

It is not the back bone of the Internet. It is the back bone of most naming services.

The internet is IP. It is very robust. DNS is an application over the top of the Internet.

Modern UDP-based protocols handle this in two ways. First, prefer to make responses no larger than the request, so there is no amplification.

Second, if the response has to be larger than the request, send the requestor an address-specific value in a small initial response, e.g. HMAC of a secret and the sender IP. Then any request that incurs a large response has to contain that value. If the sender is spoofing the IP address and can't receive the small response sent to that address, they can't cause a large response to be sent there.

This can't be done with DNS because of "security" middleboxes. They ossify the protocol because they reject anything they don't understand, and they don't understand new versions of the protocol even if both of the endpoints do. So the protocol gets frozen in time and no security improvements can be made because of the things that claim to be there to improve security.

Thanks.

> but once you get customers that can bring their own IPs (...)

You know these IPs though, since you route to your customer. So you could drop the offending packets. (I may be getting over my head here).

I know. But one layer below, there's still a chain linking back to the origin ISP. The DNS operator can ask his own ISP about it.

(Also, to people down-voting a genuine question ? wth..)

Or if you want your site to be accessible for everyone, value privacy, or remember this guy from yesterday: https://news.ycombinator.com/item?id=28552948

Disclaimer: I work at Cloudflare on Workers.

I asked around internally & Magic Transit is a solution the ISP should be able to leverage to protect themselves.

It's quite unlikely there's a single link sending the traffic; that would be super easy to block. Most likely this is used to either hide the command server or the actually compromised servers. While symmetrical load can be quite obvious, it's less so if we're talking about 2000 links sending at 20 mb/s each. Especially when those links also have legitimate traffic.

To my knowledge there's also ways to integrate network observability tools (like kentik) to automate this to a degree, for those that are big enough or DDOS events are common enough that is useful to do so.

I imagine getting paged because of a DDOS is slightly easier when it's telling you it already null routed a few IPs so the rest of your network isn't screwed and you just have to identify how problematic those specific IPs being out of service is and whether you need to take action or wait for them to get bored.

That sort of encourages the attackers, because from their point of view, the attack is succeeding while the IP is being blackholed.

I get that, but it's one of those things a lot of people go through, and blow a ton of effort on solved problems. Because it's not one of the things that's so obvious and commoditized...yet.

Peace of mind is worth a lot. If this person were with my company for example, this wouldn't even be a comment.

I say my company meaning my employer. I have exactly zero stake in mentioning them by name or advertising them.

That’s.. not what a Distributed Denial of Service does. They come from thousands of IPs.

That's the first 'D' in DDoS - distributed.

Most robots participating in this are unwitting. It's incredibly unlikely someone is paying... even with stolen funds.

Often a sort of malware that more or less sits dormant until command/control sends a target to attack

edit: To the individual machine, not a lot. The sum total is where the denial of service comes in - everything has a tipping point.

It's not expensive if you have a botnet, reflective attacks or L7 attacks. It's also not just one type of attack (overloading the target), the only thing in common is that it's a denial of service, and if you distribute it amongst many sources it's a distributed denial of service.

Distributed can mean different things, especially if you can spoof addresses, in which case you can do fake distribution (causing src based filtering to be useless). If you know it's just a single bad AS that is doing it, then you can block the AS, but if it's distributed amongst multiple AS'es that's harder. Same with reflective attacks, the source of the traffic might not be the source of the attack and might be a legitimate source which you cannot block because your internet wouldn't work anymore (i.e. when it's DNS servers or NTP servers doing the reflection, or a GCN address which would completely take an ISP's section offline).

It's not as simple as sending a large volumetric load from A to B. It used to be that way a decade ago and it can be effective today, but it's not the only way it's happening.

The attacks come from (up to) millions of different ip addresses, each one a different machine and often on a residential network.

This is the problem with botnets - all the connections look like real traffic at first glance.

The src ip can be spoofed and randomized if you don't care about getting the response (and to some degree, you care more about not getting it)

This method is basically already in use for DOS mitigation using a technique called a SYN cookie. But as the parent said, there is a huge spectrum of different ways to produce a DOS and so there is no single method of addressing them. The problem tends to be much harder as you get to higher levels, for example, requesting things from an HTTP server - there are mitigations available there as well such as Cloudflare's approach of capturing DoS like requests and subjecting them to CAPTCHA, but once again no one size fits all.

The server would still need to track some state.

Worse, the cost is another round trip delay. That's a problem for anyone with a high latency connection.

You're too lazer focused on syn attacks, there are countless others. What it boils down to is the sheer size of the botnets. You stop syn? Great, I'll make it UDP with a max payload and send it from 500k hosts. You will eventually fall over if my botnet is big enough.

> Why is this resource asymmetry unavoidable? It seems reasonable that a new protocol could solve it.

Sending a HTTP GET is easy, but the server has to process the request and send the whole page... think a few tens of bytes of a request, and a whole webpage in a reply.

The server has to do an encryption and a decryption. A legit client has to do the pre-SYN dance. In your protocol, I don’t spam pre-SYNs; I spam SYNs with fake tokens. Thus I send only the same number of packets as before, but the server has to do these decryptions! Or I can mix and match spoofed SYNs with pre-SYNs and legitimate SYNs in whatever way finds the best leverage.

Your pre-SYN design is very close to the design of TLS 1.2, by the way. That’s how I knew how to attack it. TLS is attackable in just this way: open a real connection, then lots of fake handshake requests. No crypto required to generate them, but the server has to do lots.

Now that said, BCP 38 would go a LONG way towards addressing this. There will come a day when the well-behaved networks decide they’re not accepting traffic from non-compliant peers.

> Why is this resource asymmetry unavoidable? It seems reasonable that a new protocol could solve it.

There's some sense in which a little asymmetry is unavoidable -- a malicious attacker can always just send whatever bytes they want over the wire, and a server on the other end with any practical purpose must at a minimum receive each of those bytes AND perform some operation equivalent to determining if any more work needs to be done.

To the extent that any real work or state management needs to be done by the server, the only way to avoid much asymmetry is to use a protocol forcing the client calls to be artificially more expensive before the server will agree to respond.

So a bit like a proof of work concept for packets? Doesnt sound like such a crazy idea to me

The fundamental asymmetry is bandwidth. If you can overwhelm my incoming connection, I can't do much useful work.

I've got to pay for capacity, but a bot net controller doesn't need a whole lot of bandwidth to trigger a lot. If you control 10,000 hosts with 10Mbps upload, that's 100Gbps; and botnets are growing as are typical upload bandwidths.

And that's without spoofing and amplified reflection attacks. Some of the reflection attacks have high amplification, so if you've got access to 100Gbps of spoofable traffic, you can direct Tbps of traffic.

If my server has a 10G connection and you're sending me 1Tbps, there's just no way I can work.

Syncookies work pretty well for TCP, I had run into some effective SYN floods against my hosts I managed in 2017, but upgrading to FreeBSD 11 got me to handling line rate SYNs on a 2x 1Gbps box and I didn't take the time to test 2x10G as I wasn't really seeing that many SYN floods. I don't recall any more effective SYN floods after that point. We didn't tend to have dedicated attackers though; people seemed to be testing DDoS as a service against us, and we'd tend to see exactly 90 or 300 seconds of abuse at random intervals.

Our hosting provider would null route IPs that were attacked if the volume was high enough for long enough. Null routing is kind of the best you can hope for your upstreams to do, but it's also a denial of service, so there you go.

> Why is this resource asymmetry unavoidable? It seems reasonable that a new protocol could solve it.

Protocols are trying to avoid this problem. For TCP there exist SYN cookies to reduce the amount of work a host has to do. For QUIC retry packets exist, as well as anti-amplification rules. However applying those mechanisms also has a cost. One is you still get computational cost not to zero, since you still have to receive and reject packets. And the other cost is now your legitimate users are experiencing a latency penality due to potentially requiring another round trip.

The latter means you don't want to make those mechanisms the default, but rather use them selectively in certain situations.

The former means even if you apply countermeasures you are not fully protected. If you get flooded purely by the bare amount of packets and the system stalls just from handling them, all the higher level mitigations won't work very well anymore. That's the difference between a DDoS and a DoS - the DDoS might not even need an asymetry in cost, it just brute-forces the system down.

It seems like requiring the client to provide proof of work (e.g. Hashcash[1]) could be an easy way to help reduce the asymmetry.

[1] https://en.wikipedia.org/wiki/Hashcash

Doesn't it also mean addresses serving as the source of an attack become more disposable?

if you are sending packets but don't care about the reply, you can spoof the from header.

The other form of ddos is udp amplification. you send a packet to a legit host and spoof the target ip as the from ip, the reply goes to your target and bam, ddos.

A trillion hosts.deny entries?

Most devices are configured to randomize part of the v6 address for privacy so you can no longer see more info than ipv4 would show.

The first 'D' of DDoS means distributed. Which means it is so broad. If you unplug yourself from the internet, yes you can block them... Also blackholing them like this won't be useful. Since attack is comming from multiple zombie clients (hundreds of thousands) and current internet structure is using IPv4 with CGNAT, you would block tens of millions of legitimate users at the expense of a small percentage.

Edit: Also, even though you blackhole them within your iptables, packages are still coming to you (routed) and filling up the bandwidth whether you like it or not.

And you pay your peers according to bandwidth you use. (eg. bytes in and out per month)

So, a proper solution is to use globally distributed system to broadcast specific routes so the attack gets stopped even before it reaches you. (eg. through their own ISP's route table blackholing your routes) But that means legit users from that ISP wont be able to access your network at all...

They are already doing it themselves...

How so? To me, closed-mindedness is shooting down an idea without a reason or alternative.

.ch is Swiss TLD ;)

> home router makers could intervene here and let their owners know when they’ve likely been hacked

How?

A couple of years ago I had an email from our ISP telling me that "common port X is open on your router" (forwarded to a box exactly as I set it up) and asking if I could "fix the problem".

Except it wasn't a mistake or a configuration error, I deliberately set it up that way.

I don't see how this would be feasible without giving them a backdoor to snoop around in my home network. No thanks.

That's never going to work but it very well could lead to ISPs very strongly herding customers to use their own managed routers. They already push for that for customer support purposes, using this to start automatically snooping on "malicious" traffic within a customer's network would be a big step in the wrong direction.

You could make a compromise here and require ISPs and network vendors to support a common notification protocol to identify devices sourcing malicious traffic. Most ISPs already have systems in place to send notification messages to customers that are sending botnet traffic. You could mandate it for all of them and let the customer decide if they want their router configured to automatically block a device they were notified about or just record the MAC address and send the ISP a URL the user can visit to view the device sending the traffic. You could make it friendly to the unwashed masses and still put detection on the ISPs and not give them privileged access inside every customer's network.

It wouldn't be hard.

If you can examine traffic on a network from which attacking traffic originates, you can see that it's coming from a phone.

Then you could see which app by either limiting or deleting apps one at a time, or you'd need two sources, then just see which apps the two sources have in common.

When you have 250 Gbps, it'd be simple to capture a fraction of a second's worth of traffic, then write a script to fire off emails to network admins of every IP involved and ask them to look in to it. Out of hundreds or thousands of messages, you'll get a few humans who'd look in to it and would be helpful.

I should know, because I've done this very thing.

If your nontechnical neighbour's ISP calls and says "you're participating in a DDoS, will you please find out which device behind your NAT is sending the traffic and fix whatever the problem is", do you think your neighbour can fix it in five minutes?

because nothing says you have to include your real ip in the src header of the ip packet if you don't care about the reply.

remote triggered blackholing the networks from which the ddos comes is very effective for small DDoS's. but its a very crude methods that drops entire /24's from being able to reach you.

this doesn't work for massive ddos's.

I think you understand why edpnet.be has problems now.

What they usually do is try to write packet filter rules and drop packets as close to the millions of origins as possible, plus call the networks where the sending devices are and say things like "could you please call your customers and tell them to unfuck their {phones,dsl routers,…}"

It is intrinsic to the design.

The open nature of the protocol carries with it an implicit trust. Any client may join and speak to any other client and be heard. It is a social contract, just as we have "in real life."

Imagine someone knocks on your door. You can choose to not answer, but even considering the choice costs you cognitive bandwidth. As long as you're within earshot of your door, a knock will cost you attention.

We have solutions all across the spectrum, from putting up a "no soliciting" sign, to arresting bad actors. But what if you have a riot outside your front door (a DDoS)?

The abuse system would need to detect that these ip addresses are sending a sustained amount of traffic abnormally and deny them.

Why limit it to DDoSing? It could be just one feature, but an important one.

Toolbars did just that and been used for many "evil" things..

My point exactly.

These things are easy to do in principle (most readers here can write that evil payload, or learn how to do it quickly) but there are real practical challenges to getting the code to run on millions of strangers' computers/phones/dsl routers, so most of the code that's actually deployed that widely is beneficial. Happily.

... except that they most definitely do still happen in spite of that.

But yes, they do tend to mean less obviously-just-malware apps.

Sure, the official App store can do that but that does not mean the user should not have the option to side load apps.

Getting this kind of behaviour past review is simple, and malevolent apps do happen.

Google has a clever technique for detecting them (after review) and it's good, but slow. It has a decent chance of detecting the app after a few attacks, and works by analysing which apps were installed on phones just before a factory reinstall.

> Google/Apple will catch you in hours/days

Tell that to the Hola and Honey extensions. They're known to be a botnet and for this kind of abuse for years yet you can still find them everywhere in the App/Play Store...and _so_ many tech related youtube channels keep advertising for them blindly, it's ridiculous.

Nope, they are basically right and it's a source of endless frustration to security researchers that these very simple problems are still thinks that have to be dealt with in 2021.

Usually I would agree. But not in this instance. It really is silly some isps are not doing this by default, just fixing this will reduce so many types of ddos attacks. The current size of ddos attacks are just insane. And thabks to these attacks decentralized nature of the internet is eroding everyday.

Huh?

Ingress filtering is filtering traffic coming in to your network. How are you supposed to know whether traffic coming from a certain peer actually legitimately originates from that peer?

Egress filtering is filtering traffic that exits your network. It's your network, so you should know with absolute certainty whether it's real - either the source is one of your networks, or it isn't, and if it's not, you drop it.

Please explain how ingress filtering is supposed be the better solution.

Telcos also tried claiming its impossible to secure SS7, and here we are in 2021 finally getting Stir/Shaken implemented https://www.fcc.gov/spoofed-robocalls

So what you’re saying is that legislation would help here?

It’s the zombie-botnet-machine owners who need to be running more secure software, not the botnet victims.

> it destroys reachability to your systems from the population at large

Indeed.

Sometimes I think it was never meant to be. We're allowing our computers to talk to untrustworthy strangers. It was always a matter of time before one of them sent a crafted payload...

Perhaps computers shouldn't be so reachable. Can't exploit a computer that's not even there, zero days or not.

I think I’m thinking of it reverse from you. A top 160 site could have more concurrent users than a small regional ISP.

Haven't done much traditional networking for a few years so my knowledge is not that current.

They use bgp flowspec to send out filter lists using the bgp protocol to routers, essentially a long list of ACL rules with srcip dstip dstport. These routers can filter any amount of traffic it gets in hardware, so as long as your upstream (edge transit or peering links) are not full you won't even notice the ddos.

But you need something to analyse traffic that can understand which traffic is ddos and what is normal. We used devices from Arbor which are basically regular x86 servers. These devices also received netflow so whenever a ddos target received a lot of weird traffic (lots of DNS for example) it was configured to redirect all traffic for that IP to itself where it could gain a better understanding than what you can from netflow and either filtered it locally or sent out flowspec updates to filter it on routers. You could also enable this manually for non-volumetric ddos and do some filtering, but it's hard to do if it's encrypted and if they are abusing some http call that is just expensive to handle for the servers.

ISP mitigation is usually simple. Figure out what the destination of the attack traffic is and null route[1] that, pushing those null routes upstream where possible. If the destination IPs for the abuse are too diffuse and/or the upstream won't do it on their routers where they presumably have more capacity to drop than to forward to you, you're still limited by your connection bandwidth on the connections the abuse is flowing. Depending on the use case, sometimes you can do pretty well by cycling your legitimate traffic across a large block of IPs faster than the abusers can cycle their abuse traffic, but it depends on relationships and APIs to control null routing of your upstreams. More sophisticated things like only block traffic to/from port X, or throttling rather than blocking aren't generally available (but would often be super helpful if they were).

Anti-DDoS services are quite a bit different, they generally have obscene bandwidth, and BGP advertise your attacked IPs, so they can get the traffic, then they "scrub" the traffic and provide it back to you over a tunnel (or something). Scrubbing can be relatively simple stuff like rate limiting UDP and especially fragmented UDP (which is usually stuff coming from reflection attacks) or more complex stuff like looking at TCP options to determine good vs bad traffic and only letting good traffic through, or tracking TCP SYNs and only letting retries through, plenty of other stuff with TCP state machines (or half state machines, if they're only seeing the incoming traffic and not the outgoing traffic). Oh, and they usually charge you gobs of money.

[1] if destination == IP, drop packet

Traceroutes. Peering information is also often public.

More sinister options like insider information or leaked/exfiltrated network plans are also an option, but usually you get pretty far with standard network diagnostic tools.

Depends what kind of attack they use. A few years back, reflection attacks were all the rage, and it was cheap as fuck. But I can't say how many Gbps they delivered.

They'd usually market this as "stresstest" service, and you could pay by Paypal etc. Curiously they were all behind a Cloudflare-anti-DDoS wall. I still wonder if Cloudflare knew/knows about them, probably good for their business.

Then you might want to check sonicwall DPI.

Huh? I wrote this:

> Same with the DNS resolvers: if someone has an open DNS resolver that is constantly being abused, as all of them are, just cut them off right? [etc.]

That's what I'm not understanding why it is not being applied to easily solve those reflective DDoS attacks.