My naive take on a Cloudflare perspective wants to combine "three times is enemy action" with toddler-speed block dropping and manual clearance. What's the money reason this problem isn't handled at the ISP level?
The distributed nature of DDoS means each participating host isn't sending that much traffic, and there are often tens or hundreds of thousands of participating hosts. An ISP should verify claims of abuse before cutting off customers, and since most of the customers are presumably unaware of what their systems are doing, there will be a lot of unhappy customers and then you've got to spend a lot of customer support time on helping them clean up their systems so they can get back online.
I spent a fair amount of time sending out abuse reports for phishing / malware senders about a decade ago, and most abuse reporting addresses are a black hole. Even if you do get to someone who will do something about abuse, they won't do it quickly.
There's be a few high profile longer term DDoS attacks lately, but when I was running infra that got a lot of stuff, it was mostly people kicking the tires on DDoS as a service offerings and most attacks were 90 seconds long ... there's no way I'm convincing an ISP to drop a pwned customer over that.
Starting from there, this DDoS sounds like layer 7 DDoS which is easy to track to the immediate senders, but a ton of DDoS is volumetric stuff, often volumetric reflection attacks where the senders spoof your address. If you're getting that, best you can do is get the reflectors kicked off (or cleaned up) ... tracing back to the sending hosts means getting a reflector (and their ISPs) engaged to do a lot of labor intensive work.
All of that investigation stuff takes qualified people lots of time, that's your money reason it doesn't happen.
Every victim (such as readthedocs), or even people sharing blocklists to avoid becoming a victim, blocking that ISP's ranges until they do clean up their network could be a convincing argument?
As you say, even at 90 seconds, it's clear to all involved parties that the customer is pwned or malicious. Such a reoccurring source of abuse needs to either clean up or find themselves a different ISP to spread harm onto the net
I get what you're saying about that this won't solve an ongoing attack right this minute, or even by next week. But if we just let it all happen then the solution is going to be either (1) we all buy equipment that can handle something like a terabit per second and arm our infrastructure to the teeth or (2) centralize all traffic through a vetting entity who decides which client gets to visit the internet today. So far we're headed towards the latter and nobody really wants that. Abuse messages will have to slowly trickle down from victims to originating ISPs to users, and if users didn't willingly sign up, then to wherever users are getting this malware (Google's app store will be a big component). Stopping this at the source seems to me a much more desirable long-term solution
The report makes it pretty clear that the attack was distributed enough that profiling for blocklists was ineffective.
It sure is --- but an ISP would want to observe the traffic themselves, and if it's a 90 second attack every so often, chances are they won't see it when they look. When it's volumetric reflection, you can probably tell them how to send a request and see the response, and maybe they'll contact the customer, but maybe they'll just sit on it. As a victim, the ROI for reporting just wasn't there.
I wasn't getting huge traffic flows, and I was mostly getting attacks against www, which wasn't my actual service, so making sure volumetric attacks below my interface rate were shrugged off and taking simple actions like dropping requests from http clients with user-agent Wordpress were good enough. If the volumetric attacks were much over 10G, my host would have null routed my servers, which is annoying but highly scalable --- many ISPs support a BGP blackhole community, so my host can add my attacked IP to that and their upstreams will drop inbound packets when they enter the ISPs network.
I can't find a reference now, but I've seen things that allowed for more specific blackholing, such as by source or destination port number or by protocol. If my host's ISPs are dropping all UDP and IP fragments to my IP under attack, I could keep serving my TCP traffic and ignore a huge DDoS. I wouldn't even be able to measure the size of the DDoS.
My ISP didn't, when they got access logs from a service I attacked as a teenager and asked me to explain that to get the connection unblocked
Idk, at the moment we're simply not even trying to set a standard. Maybe it would work reasonably well when the ISP needs to observe the traffic and, after a few days of the initial report, they observe a netflow that matches a new abuse report. Even if we set low standards, currently, too few people are sending abuse notifications instead of just sticking it behind the great internet vetting service and calling it good
> maybe they'll contact the customer, but maybe they'll just sit on it.
That's the core point no? If they don't care about their abusive traffic, nullroute their ranges. If admins consistently do that, the abuse has to stop or the ISP goes out of business
It's a double-edged sword. That is how you end up with most ISPs blocking port 25 completely. If your hosting provider is on the list you are collateral damage. You yourself can do very little to remedy the situation except to beg your provider "to look into it".
What do you expect the ISPs to do in this story? We are talking about TSL connections. Block port 80 and 443 and expect the costumers to use your HTTPS-Proxy. Than they could inspect and block individual actions.
Yes, and this sucks. I moved ISPs because the original one had burned IP addresses that you can't send email from. Every ISP that gets the ranges burned like that will eventually either goes out of business or gets their act together. Not by tomorrow, but eventually
Like, the only other alternative outcome I see is that everyone has to pass through a central surveillance point that decides who's benign and who's naughty, and since nobody wants that... what else are we to do but report abuse?
> What do you expect the ISPs to do in this story? We are talking about TLS connections.
What they've done to me when I abused a service as a teenager, the ISP got an abuse notification: cut off the connection, ask the subscriber wtf this traffic is and how they're going to make sure it doesn't happen again (at which point my dad, the subscriber, came to me and asked if I knew something about this.... yeah ^^')
TLS doesn't matter because the ones receiving the abusive traffic can say what it was. Their access logs will contain the decrypted information and that should be put in an abuse report. If the customer denies everything, turn on netflow logging for a month and see if a future abuse report comes in that can be correlated against the netflow logs. Or have netflow logs stored for 1h by default and retain the entries for which an abuse notification came in (grep for IP addresses in incoming notifications). Lots of reasonable options there; this isn't the difficulty. It's getting people to send that abuse notification so the ISP can identify the problematic subscribers
I saw this quite a few years ago. It is possibly relevant. I don't know what, if anything, came of it since then.