One starts to wonder, at what point might it be actually feasible to do it the other way around, by whitelisting IP ranges. I could see this happening as a community effort, similar to adblocker list curation etc.
Source: stopping attacks that involve thousands of IPs at my work.
Are you really? How likely do you think is a legit customer/user to be on the same IP as a residential proxy? Sure residential IPS get reused, but you can handle that by making the block last 6-8 hours, or a day or two.
My single-layer thought process:
If they're knowingly running a residential proxy then they'll likely know "the cost of doing business". If they're unknowingly running a residential proxy then blocking them might be a good way for them to find out they're unknowingly running a residential proxy and get their systems deloused.
And what if I'm behind CGNAT? You will block my entire ISP or city all in one go, and get complaints from a lot of people.
Alas, the "enough users get annoyed by being blocked and switch ISPs" step will never happen. Most users only care about the big web properties, and those have the resources to absorb such crawler traffic so they won't get in on the ISP-blocking scheme.
But my main point was in the second paragraph, that "enough of them would" will never happen anyway when the only ones doing the blocking are small websites.
What, exactly, do you want ISPs to do to police their users from earning $10 of cryptocurrency a month, or even worse, from playing free mobile games? Neither one breaks the law btw. Neither one is even detectable. (Not even by the target website! They're just guessing too)
There are also enough websites that nobody is quitting the internet just because they can't get Netflix. They might subscribe to a different steaming service, or take up torrenting. They'll still keep the internet because it has enough other uses, like Facebook. Switching to a different ISP won't help because it will be every ISP because, as I already said, there's nothing the ISP can do about it. Which, on the other hand, means Netflix would ban every ISP and have zero customers left. Probably not a good business decision.
You seem to think I said users will think the block is initiated by the ISP and not the website. I said no such thing so I'm not sure where you got this idea.
>What, exactly, do you want ISPs to do
Respond to abuse reports.
>Neither one is even detectable. (Not even by the target website! They're just guessing too)
TFA has IP addresses.
>Which, on the other hand, means Netflix would ban every ISP and have zero customers left.
It's almost like I already said, twice even, that the plan won't work because the big web properties won't be in on it.
You can be completely forgiven if you're speaking from a non-US perspective, but this made me laugh pretty hard -- in this country we usually have a maximum of one broadband ISP available from any one address.
A small fraction of a few of the most populous, mostly East-coast, cities, have fiber and a highly asymmetrical DOCSIS cable option. The rest of the country generally has the cable option (if suburban or higher density) and possibly a complete joke of ADSL (like 6-12Mbps down).
There is nearly zero competition, most customers can choose to either keep their current ISP or switch to something with far worse speed/bandwidth caps/latency, such as cellular internet, or satellite.
- Blacklisted IP (Google Cloud, AWS, etc), those were always blocked
- Untrusted IPs (residential IPs) were given some leeway, but quickly got to 429 if they started querying too much
- Whitelisted IPs (IPV4 addresses are used legitimately by many people), for example, my current data plan tells me my IP is from 5 states over, so anything behind a CGNAT.
You can probably guess what happens next. Most scrapers were thrown out, but the largest ones just got a modem device farm and ate the cost. They successfully prevented most users from scraping locally, but were quickly beaten by companies profiting from scraping.
I think this was one of many bad decisions Pokémon Go made. Some casual players dropped because they didn't want to play without a map, while the hardcore players started paying for scraping, which hammered their servers even more.
The known good list is IPs and ranges I know are good. The known bad list is specific bad actors. The data center networks list is updated periodically based on a list of ASNs belonging to data centers.
There are a lot of problems with using ASNs, even for well-known data center operators. First, they update so often. Second, they often include massive subnets like /13(!), which can apparently overlap with routes announced by other networks, causing false positives. Third, I had been merging networks (to avoid overlaps causing problems in nginx) with something like https://github.com/projectdiscovery/mapcidr but found that it also caused larger overlaps that introduced false positives from adjacent networks where apparently some legitimate users are. Lastly, I had seen suspicious traffic from data center operators like CATO Networks Ltd and ZScaler that are some kind of enterprise security products that route clients through their clouds. Blocking those resulted in some angry users in places I didn't expect...
And none of the accounts for the residential ISPs that bots use to appear like legitimate users https://www.trendmicro.com/vinfo/us/security/news/vulnerabil....
It should be illegal, at least for companies that still charge me while I’m abroad and don’t offer me any other way of canceling service or getting support.
Would it make sense to have a class of ISPs that didn't peer with these "bad" network participants?
Not sure what my point is here tbh. The internet sucks and I don't have a solution
Say you whitelist an address/range and some systems detect "bad things". Now what? You remove that address/range from whitelist? Doo you distribute the removal to your peers? Do you communicate removal to the owner of unwhitelisted address/range? How does owner communicate dealing with the issue back? What if the owner of the range is hosting provider where they don't proactively control the content hosted, yet have robust anti-abuse mechanisms in place? And so on.
Whitelist-only is a huge can of worms and whitelists works best with trusted partner you can maintain out-of-band communication with. Similarly blacklists work best with trusted partners, however to determine addresses/ranges that are more trouble than they are worth. And somewhere in the middle are grey zone addresses, e.g. ranges assigned to ISPs with CGNATs: you just cannot reliably label an individual address or even a range of addresses as strictly troublesome or strictly trustworthy by default.
Implement blacklists on known bad actors, e.g. the whole of China and Russia, maybe even cloud providers. Implement whitelists for ranges you explicitly trust to have robust anti-abuse mechanisms, e.g. corporations with strictly internal hosts.
Google indexes in country, as does a few other search engines..
Would recommend.
I looked at all the IP ranges delegated by APNIC, along with every local ISP that I could find, unioned this with
https://lite.ip2location.com/australia-ip-address-ranges
And so far i've not had any complaints. and I think that I have most of them.
At some time in the future, i'll start including https://github.com/ebrasha/cidr-ip-ranges-by-country
Cloudflare has been a godsend for protecting my crusty old forum from this malicious, wasteful behavior.