Hacker News Reader

Comment by ineedasername | Hacker News Reader

I'm very much not a network engineer, but I'd like to understand the magnitude of this issue because my intuition is wrong:

250 Gbps seems like it would definitely be a lot for a server or website but it also seem like a drop in the bucket for an ISP providing broadband for many customers.

Clearly I'm wrong because it is an issue here. I'd like to understand why I'm wrong, and I hope that here, on HN, that's taken in the spirit of curiosity intended and not negativism.

So, what am I missing? Maybe Belgian broadband is lower capacity than what I'm used to in a US metropolitan area? Maybe this particular ISP served a population too small to have a... um... "fat pipe"? I'd like to understand.

View on HN

An ISP has two types of edge connections where ddos traffic comes into their network. Private peering where they are directly connected to their neighbor in one or more locations, typically neither party pays for traffic, can be 1 to 100G interfaces and multiple of them. Second type is transit, smaller or mid sized buys it from larger ISPs and pay for traffic. Everything that does not go directly to a peer mentioned before goes over this connection. Can also be very different in size.

You can just add upp all edge Links and know their theoretical limit, but most of the ddos traffic likely comes from Asia and it's unlikely that they peer directly with those ISPs. So most of the traffic will go over transit, but the larger ISP likely has an automated system that handles volumetric attacks. So if there's any impact it will be on transit mostly affecting traffic to/from outside of Europe. 250 might be a lot for this ISP, but they might also have a lot more transit headroom, or it light be handled by their transit provider. Who knows.

As an outsider to network topology that's fascinating to understand. Thanks for explaining.

If you don't mind going further, how does an ISP (or anti-DDoS services) mitigate something like this without incurring even more overhead to examine the traffic & not route it onwards? Just a flat block against the sources, or something more sophisticated?

Haven't done much traditional networking for a few years so my knowledge is not that current.

They use bgp flowspec to send out filter lists using the bgp protocol to routers, essentially a long list of ACL rules with srcip dstip dstport. These routers can filter any amount of traffic it gets in hardware, so as long as your upstream (edge transit or peering links) are not full you won't even notice the ddos.

But you need something to analyse traffic that can understand which traffic is ddos and what is normal. We used devices from Arbor which are basically regular x86 servers. These devices also received netflow so whenever a ddos target received a lot of weird traffic (lots of DNS for example) it was configured to redirect all traffic for that IP to itself where it could gain a better understanding than what you can from netflow and either filtered it locally or sent out flowspec updates to filter it on routers. You could also enable this manually for non-volumetric ddos and do some filtering, but it's hard to do if it's encrypted and if they are abusing some http call that is just expensive to handle for the servers.

ISP mitigation is usually simple. Figure out what the destination of the attack traffic is and null route[1] that, pushing those null routes upstream where possible. If the destination IPs for the abuse are too diffuse and/or the upstream won't do it on their routers where they presumably have more capacity to drop than to forward to you, you're still limited by your connection bandwidth on the connections the abuse is flowing. Depending on the use case, sometimes you can do pretty well by cycling your legitimate traffic across a large block of IPs faster than the abusers can cycle their abuse traffic, but it depends on relationships and APIs to control null routing of your upstreams. More sophisticated things like only block traffic to/from port X, or throttling rather than blocking aren't generally available (but would often be super helpful if they were).

Anti-DDoS services are quite a bit different, they generally have obscene bandwidth, and BGP advertise your attacked IPs, so they can get the traffic, then they "scrub" the traffic and provide it back to you over a tunnel (or something). Scrubbing can be relatively simple stuff like rate limiting UDP and especially fragmented UDP (which is usually stuff coming from reflection attacks) or more complex stuff like looking at TCP options to determine good vs bad traffic and only letting good traffic through, or tracking TCP SYNs and only letting retries through, plenty of other stuff with TCP state machines (or half state machines, if they're only seeing the incoming traffic and not the outgoing traffic). Oh, and they usually charge you gobs of money.

[1] if destination == IP, drop packet

It's only simple if you don't care that the service behind that IP is down. I would only use it as a last resort if there are no others tools available or if i was a consumer ISP. Any competent ISP with business customers, hosting or otherwise, wouldn't be doing this.

Null routing is often (but not always) used in the scenario that the service behind the IP is already down, but null routing will improve service to everyone else, so it's an acceptable casualty.

I was (working for) a major customer of a competent hosting service and they'd do null routes (automated) on our service IPs when we got attacked. Of course we complained, but it was that or have automation put us behind a traffic 'scrubber' that caused more problems than null routing. In the end, what made it right for us was when they got the traffic levels and sampling intervals set properly; only null route if incoming traffic is above line rate for the server, and initially null route for a short time (5 minutes, I think) before resampling and escalating the time. Most attacks were short, so the default of 6 hours was excessive.

And if you just handled the ddos by dropped the correct traffic the service would be unaffected.

I mean, yes, but a) they didn't have the equipment that could do that at the rate required, b) they certainly couldn't ask their upstreams to do that and at least for some of the attacks, the attack traffic added enough traffic to go over capacity of some of their links; having the upstream null route it was possible and solved the congestion.

ISPs have a very decentralized network, that is, they have routers in a lot of places.

If you sum up the available traffic capacity, you get a number that's far bigger than 250 Gbit/s.

But that's not the right comparison to make. Much of that DDoS traffic goes through multiple of the ISP's routers, and if you're doing just a numbers comparison, you'd have to multiply the attack rate by the (average) number of hops.

But things are even more complicated in reality: many smaller links will have only 10Gbit/s or 20Gbit/s or 40Gbit/s capacity (or maybe even 1G), and saturating them not only causes increased latency, but also routing the traffic around causes more overall load on the network.

Then there are management systems behind firewalls, and if you DDoS them, it's typically the firewalls that give out first.

Finally, customers don't get the full throughput internally available to an ISP; if you DDoS a customer's IP, their link will be saturated, even if the ISP itself has plenty to spare.

(Not a network engineer myself, but working closely with some).

Wouldn't that all be true of normal web traffic too, presumably already capable of rates beyond 250Gbps? Though they may not have that much free headroom on top of normal traffic.

So in the case where the attackers are aiming their traffic at one of the ISP's routers to saturate a link, how do they get the IP address/domain name of this router?

Traceroutes. Peering information is also often public.

More sinister options like insider information or leaked/exfiltrated network plans are also an option, but usually you get pretty far with standard network diagnostic tools.

250Gbps is still a lot of bandwidth for a network even today. An enterprise ISP like Verizon or Spectrum may have less of a capacity issue with it but for many other networks this is a lot.

Quick example, though not an ISP, but the entire Internet archive operates on 60Gbps of bandwidth. So an attack of 250 would be quite a problem.

http://blog.archive.org/2020/05/11/thank-you-for-helping-us-...

the entire Internet archive operates on 60Gbps of bandwidth

Wow, yes, that definitely puts the magnitude of this into perspective. Thank you.

Though that's a substantial amount of traffic: isn't this currently going through one ISP, which (I would hope) has many more customers than just archive.org? "top 160 site" is significant, but still minuscule compared to all internet traffic.

I think I’m thinking of it reverse from you. A top 160 site could have more concurrent users than a small regional ISP.

It might! I suspect my intuition fails at this scale. I have absolutely no data, so I'd definitely be interested in any that can be found.

In my defense though! 60Mbps is literally less than one billionth the estimated internet bandwidth these days: https://en.m.wikipedia.org/wiki/Internet_traffic . I doubt there are even one million ISPs, so it still seems relatively "small" to me. It could definitely be bigger than [random company X] can handle, but it seems like it stands a good chance of being smaller than most in aggregate, so even if it's hard for X to support it's not particularly "interesting" in a state-of-the-art way.

And I can completely believe I'm wildly wrong in this. 999/1000+ low-technical-effort ISPs doesn't seem unlikely to me, so 60Mbps could very well be a major achievement as far as I know. I'd be surprised, but... I've been surprised enough that I'm no longer surprised at being surprised. Humans are weird at scale.

The Wikipedia entry you linked to estimates 2021 internet traffic at 276 EB/month. That’s an average 839,625 gigabits per second. So 60 Gbps is one 14,000th of that.

Of course, one figure was about capacity and the other about actual traffic, but still.

Ah, per month. Yep, I missed that, I thought it was per second.

I dunno, my home operates on 2Gbps of bandwidth. 250Gbps is half of our neighborhood (not sure what the shared uplink size is though).

It's not just 250Gbps, its 250 on top of normal usage. Most ISPs run a fairly efficient setup where they are only paying for a little bit over the average usage.

For comparison, the entire internet averages on the order of 50 Tbps*, so this is about 0.5% of that. Residential internet might have a max rate of 1Gbps in the best conditions but if you use even 1% of that consistently you'll get overage fees. Steady bandwidth numbers are typically much lower than max bandwidth numbers.

* Based on 122,000 PB/month in 2017 from https://en.wikipedia.org/wiki/Internet_traffic, if my calcs are right

I don’t know anything about this particular isp or attack, but if that 250 gbps is aimed at specific parts of their infrastructure that are not designed for this much load, it’s likely to saturate individual links, servers, and/or network gear.

Their announcement said “all of our services”, so this could be anything from their authoritative/recursive dns to monitoring or billing servers to specific routing nodes.

So even if they had nice fat 100G pipes coming in, and those pipes weren’t saturated, a ddos of this size with a mix of vectors can be debilitating on enough discrete pieces of infrastructure that the isp is effectively “down” for many end users.

(Again I am only speaking generally with guesses about what might be going on at this isp. I’m also not a network engineer but I do write code for network gear that does anti ddos stuff.)

Reply on news.ycombinator.com