An aggressive, stealthy web spider operating from Microsoft IP space
utcc.utoronto.ca
utcc.utoronto.ca
Just yesterday I put all their networks into an nginx geo map:
geo $limit_bots_ip {
# requests with an empty key are not evaluated by limit_req
# see: http://nginx.org/en/docs/http/ngx_http_limit_req_module.html
default '';
157.55.39.0/24 'bot';
207.46.13.0/24 'bot';
40.77.167.0/24 'bot';
13.66.139.0/24 'bot';
...
}
limit_req_zone $limit_bots_ip zone=badbots_ip:1m rate=1r/m;
I suspect this is a real Bing web crawler that's possibly misconfigured. The "spoofed" user-agent might be an attempt to get a more genuine crawl of what a mobile browser sees. I at least know Google does this.
This particular 'bot' would start crawling in-line Javascript on a customer site, and then get stuck in an infinite loop requesting things like:
/page/window.open
/page/window.open/page/window.open
/page/window.open/page/window.open/page/window.open
Unfortunately, we do see enough legitimate requests with that UA that we had to resort to the IP blocking. A bunch of /16's and a few smaller ranges... it feels dirty :(
> Given that there are websites that are willing (or reluctantly forced) to allow Google(bot) access but would rather like to block everyone else, more than a few of them are probably using User-Agent matching instead of anything more sophisticated. (https://utcc.utoronto.ca/~cks/space/blog/web/SpiderUserAgent...)
Yeah, what a great idea to make all of our websites only crawlable by Google. Because competition in the search engine space is _so_ undesirable.
It wasn't so much a "ok, I want only google to search this and not anyone else", but rather at the time, there wasn't anyone else (that behaved).
So now, you've got a website that is only searchable by google, and the webmaster retired and you've got this robots.txt that says you can't do it - and even some active defenses so that when non-google hits the page, they get tarpited. What are competing (reasonably well behaved) web spiders to do? How does Bing get in there to be able to compete now?
Its not that Bing should or shouldn't be doing this - but there's a fairly reasonably way that we got to the point where are now without any nefarious behavior from the various participants.
User-Agent: *
Disallow /
User-Agent: Googlebot
Allow /
Do I blame Bing for that? Absolutely not.
Having a silly robots.txt is one thing- going to the lengths of blocking all bots that aren't in Google's IP space is what I'm worried about. If the whole world did that, nothing but Google could survive.
Now roles are reversed (or at least, equal - though if DDG consuming bing data is considered to be on the side of good, an argument could be made for reversed) and yet those configurations persist because they were forgotten about.
My post above is a just so story ( https://en.wikipedia.org/wiki/Just-so_story ) for why they are the way they are now. Old systems that work tend to be forgotten until a security fix needs to go in and Chesterton's fences in config files tend to be left.
With todays resources, I would not advocate indiscriminate tcp tarpits for things that might be badly behaved crawlers until they demonstrate that they are indeed badly behaved crawlers or scanners searching for exploits.
…something which Google has never done, I'm sure.
shagie brought up the possibility that somebody would have a robots.txt with only Google in it because when they wrote their robots.txt, they wanted to block poorly behaved / evil crawlers and Google's was well-behaved. But that doesn't make sense to me, because by definition poorly behaved / evil crawlers do not care about robots.txt.
The thing was that you'd send everything that was requesting faster than some rate limit to that tarpit unless it was google.
Google, Bing, Archive.org, and people normally browsing were just fine while bad crawlers got a blank page.
I would hope they did investigate too otherwise it’s too easy to DoS a valid customer with fake complaints.
AWS has acted upon similar reports in the past, Azure may have but the lack of comms and delays in seeing any action (if any) hasn't inspired confidence.
A couple provides I sent abuse complaints to based on whois lookups responded with automated emails requesting the info in X-ARF format. It’s hard to get some cloud providers to respond to a random residential user.
A couple IPs were repeated “scans” by supposed security researchers with IPs out of Hawaii and France.
If I had a dollar for every person who has turned loose such a tool to shit up my abuse@ispname.com inbox...
I seriously doubt the Las Vegas “located” IP with Hong Kong addresses on their registration or Shanghai UCloud Information Technology Company IPs are scanning me for good reasons.
What do you think would happen if I run an older version of plex with a known vulnerability?
The automated emails are usually in the same formats making them easy to filter and parse too.
If I find the sketchy Hawaii and French IPs I will post something.
Edit: Here you go:
https://academyforinternetresearch.org
IP Location United States United States Honolulu Academy Of Internet Research Limited Liability Company
ASN United States AS400161 (registered Oct 22, 2021)
Whois Server whois.arin.net
IP Address 104.156.155.3
NetRange: 104.156.155.0 - 104.156.155.255
CIDR: 104.156.155.0/24
NetName: ACDRESEARCH
NetHandle: NET-104-156-155-0-1
Parent: NET104 (NET-104-0-0-0-0)
NetType: Direct Allocation
OriginAS:
Organization: Academy of Internet Research Limited Liability Company (AIRLL)
RegDate: 2022-01-07
Updated: 2022-01-07
Ref: https://rdap.arin.net/registry/ip/104.156.155.0
OrgName: Academy of Internet Research Limited Liability Company
OrgId: AIRLL
Address: #A1- 5436
Address: 1110 Nuuanu Ave
City: Honolulu
StateProv: HI
PostalCode: 96817
Country: US
RegDate: 2021-10-15
Updated: 2022-11-06
Ref: https://rdap.arin.net/registry/entity/AIRLLIt seems they are running on Hivelocity a bare metal hosting provider.
Snort is something I have looked at installing and playing with on a PFSense box.
You could also check if the remote IP has any open ports (ex. SSH) and block based on that as most regular visitors probably won't be exposing those types of services to the Internet.
Most bots are using really old libraries. Some of the newer bots are using farms of cellular cards but they can be spotted by the use of TCP Timestamps as that is enabled by default on Android. The downside of doing anything with this is that one would also block legit cell phones, so just give them a simple puzzle to solve. e.g. What is 1+4 or if you are certain they are a bot then ask How many roads must a man walk down?
People who are doing a 'get all content on the web' scrape, I try to not block, because there are plenty of reasons to want an offline copy of the internet.
I try to make sure all my webapps have sensible URL schemes to prevent such (IMO legitimate) activities ending up doing an infinite crawl because for example the URL has a session identifier in it and the bot keeps getting assigned a new session.
That's what I saw from Azure IPs recently. A single URL being requested from some 5k IPs, each one requesting it every few minutes.
I had reported it to MS, they said (after two weeks) "it looks like abuse" and they'd forward it to their cert. I haven't heard any update since.
It was a really dumb spider. It wasted a huge amount of its efforts requesting pages that are no longer valid on my site. It must have had an archived set of URLs it was using as a source rather than spidering the site as it is today.
Generating 404 responses puts a considerable load on WordPress sites and generates a lot of network traffic, but these have been relatively easy to block because of the predictable user agent and URI path prefixes. I'm thinking about blocking the Azure ASN completely, or developing something akin to Cloudflare's "are you a human?" interstitial page when requests come from cloud provider ASNs.
Started around Jan 12. Large pool of IP addresses, hard to block. Occasional brief DOS impacts but mostly just annoying errors in my logs. (if too many crop up we get automatic alerts). What was really puzzling is that many of the URLs are old (e.g. request for details on hosted sites that no longer exist). I loaded up a 6 month old backup database and confirmed those accounts weren't present, so the source list of URLs must be older that. Really bizarre.
After reading this article I looked and confirmed via spot checks they are from Microsoft IPs and Safari 15.1.
If we don’t start batting an eye on this soon it’ll be the new normal that webmasters/content producers don’t really need readers or income.
There's already way more content being produced out there (some of which is done altruistically without any expectation of income) than there is demand for it that making even more of it may not be as valuable as you think.
Fast forward a few years and you have no more audience, no one has to know you exist, you ghost write for a magic black box that people ask questions and throw money at. When you want to impress a new friend and mention you published some interesting niche research (e.g., how to repair a thing) they skeptically nod because in their head no one does it anymore, the black box has all the answers.
Will you still be feeling motivated to altruistically share content tomorrow?
Never see it coming.
The list provided in this comment [0] looks much like it.
The crawls came from Azure and AWS. Forged UAs, repeat hits in the same URL, etc.
Some examples of what I found you can't do at perhaps 20% of business websites:
* Use the website near a country border where people often commute freely across borders on a daily or regular basis (example: Schengen region).
* Plan a holiday figuring out when a shop closes for the day in another country.
* Order something from an Internet connection in another country to arrive when you return home.
* Look up product information whilst on a work trip overseas.
* Access the website via a workplace proxy server that is mandated to be used and just so happens to be hosted in a data centre (the likes of AWS and Azure included) which is blocked simply because it's not a residential ISP. A more complex check would be the website checking the time between a request from Javascript code executing on client reaching the server matches the time the server thinks it should have taken (by ICMP PING or similar to the origin IP address).
* Use the website simply because the geo IP database the website uses hasn't been updated to reflect block reassignments by the regional registry, and thinks that your French residential IP address is a defunct Latvian business IP address.
* Find the business on a price comparison website.
* Find the business on a search engine that isn't Google.
* Access the website without first allowing obfuscated Javascript to execute (example: [1]).
* Use the website if you had certain disabilities.
* Access the website with IPv6 or using 464XLAT (shared origin IPv4 address with potentially a large pool of other users).
The answer to me appears obvious: Store and product information is for the most part static and can be stored in static HTML and JSON/GeoJSON files that are rewritten on a regular cycle, or at least cached until their next update. JSON files can be advertised freely to the public so if anyone was trying to obtain information off the website, they could do so with a single request for a small file causing as minimal impact as possible. It's not difficult to create a website where 10,000 requests per second can be made to static data like product information or store locations. More advanced features such as stock availability would cause additional load, but again, 10,000 simple queries to a relational database is not a challenging problem.
Of course, none of the anti-bot measures implemented by websites actually stop bots they're most seeking to block. There are services specifically designed for scraping websites that have 100,000's of residential IP addresses in their pools around the world (fixed and mobile ISPs), and it's trivial with software such as selenium-stealth to make each request look legitimate from a Javascript perspective as if the request was from a Chrome browser running on Windows 10 with a single 2160p screen, etc. If you force bots down the path of working around anti-bot measures, it's a never ending battle the website will ultimately lose because the website will end up blocking legitimate customers and have extremely poor performance that is worsened by forcing bots to make 10x the number of requests to the website just to look legitimate or pass client tests.
[1] https://www.imperva.com/products/web-application-firewall-wa...
We used a combination of ASN and UA to block them.
"/var/www/domains/www.example.org/wp"
"/var/www/domains/www.example.org/bc"
"/var/www/domains/www.example.org/bk"
"/var/www/domains/www.example.org/backup"
"/var/www/domains/www.example.org/old"
"/var/www/domains/www.example.org/new"
"/var/www/domains/www.example.org/main"
"/var/www/domains/www.example.org/home"
"/var/www/domains/www.example.org/Telerik.Web.UI.WebResource.axd"
"/var/www/domains/www.example.org/remote/fgt_lang"
"/var/www/domains/www.example.org/wordpress"
"/var/www/domains/www.example.org/Wordpress"
"/var/www/domains/www.example.org/WORDPRESS"
"/var/www/domains/www.example.org/WordPress"
"/var/www/domains/www.example.org/wp"
"/var/www/domains/www.example.org/Wp"
"/var/www/domains/www.example.org/WP"
"/var/www/domains/www.example.org/old"
"/var/www/domains/www.example.org/Old"
"/var/www/domains/www.example.org/OLD"
"/var/www/domains/www.example.org/oldsite"
"/var/www/domains/www.example.org/new"
"/var/www/domains/www.example.org/New"
"/var/www/domains/www.example.org/NEW"
"/var/www/domains/www.example.org/wp-old"
"/var/www/domains/www.example.org/2022"
"/var/www/domains/www.example.org/2020"
"/var/www/domains/www.example.org/2019"
"/var/www/domains/www.example.org/2018"
"/var/www/domains/www.example.org/backup"
"/var/www/domains/www.example.org/test"
"/var/www/domains/www.example.org/Test"
"/var/www/domains/www.example.org/TEST"
"/var/www/domains/www.example.org/demo"
"/var/www/domains/www.example.org/bc"
"/var/www/domains/www.example.org/www"
"/var/www/domains/www.example.org/WWW"
"/var/www/domains/www.example.org/Www"
"/var/www/domains/www.example.org/2021"
"/var/www/domains/www.example.org/main"
"/var/www/domains/www.example.org/old-site"
"/var/www/domains/www.example.org/bk"
"/var/www/domains/www.example.org/Backup"
"/var/www/domains/www.example.org/BACKUP"
"/var/www/domains/www.example.org/SHOP"
"/var/www/domains/www.example.org/Shop"
"/var/www/domains/www.example.org/shop"
"/var/www/domains/www.example.org/bak"
"/var/www/domains/www.example.org/sitio"
"/var/www/domains/www.example.org/bac"
"/var/www/domains/www.example.org/sito"
"/var/www/domains/www.example.org/site"
"/var/www/domains/www.example.org/Site"
"/var/www/domains/www.example.org/SITE"
"/var/www/domains/www.example.org/blog"
"/var/www/domains/www.example.org/BLOG"
"/var/www/domains/www.example.org/Blog"
From an anthropological point of view one of the most fascinating behaviors here is the apparent belief that as a website operator, your opinions about how HTTP clients should craft their user agent headers have any weight whatsoever.
This curious belief system predates even the browser wars let alone the modern web. Remarkable to find an original example of this idealistic world view still being expounded today.
Not caring about false useragents is one thing, caring about somebody running a multi-thousand-requests-per-minute spider requesting pages that no longer exist is another thing. It would still be valid cause for complaint even if it had a valid useragent.
The internet would be a better place if more experienced people (such as those who were running httpd 22-24 years ago) reinforced and emphasized the principle, when training and mentoring younger people, that being a good citizen with your network traffic is the responsible and ethical thing to do.
I get that morally we might feel like it is better for these ill-mannered varmints to receive a well-deserved '401 Unauthorized' HTTP response when they try their shenanigans.
But... really, if they were just going to get a '404 Not found' anyway, who cares? Your server has to serve pretty much the same number of bytes, and it has to run less regexes over user agent headers.
There's just something I find charmingly anachronistic about trying to operate a server like this.
One site being hammered is one thing. Thousands of sites being hammered at the same time is another. The issue in the article is a current and live problem. Blocking the requests saves a lot of CPU cycles.
I’m just astonished that there are still people who have the time to notice and lament such behavior, and devote time to handcrafting .htaccess rules, rather than just having an aggressive set of heuristics and accepting that some crap will still get through.
The idea that you can appeal to the community to do a better job of policing badly behaved clients on their networks just feels like a quaint relic of the arpanet.
Have you ever worked in network engineering for an ISP that holds a decent sized chunk of ARIN or RIPE ipv4 IP space, and actually cares about its long term IP space reputation?
There's a big difference between a $5/mo colo/VPS operator that doesn't give a damn, and an ISP that does pay attention when legit "you have customers with abusive traffic" reports reach their network engineering team.
Your comment about not caring about policing badly behaved clients on a network seems like the sort of organization that would land itself on every SMTP RBL in existence.
A hosting company (and Azure and AWS are just the giant corporate evolution of a hosting/colo company) needs to be prepared and have a process to fire a customer for violation of TOS/AUP. It's done all the time. The immense size and callous nature of Microsoft/Azure makes this less likely in the modern era.
What’s the equivalent mechanism whereby abusive HTTP traffic gets identified and causes an ISP to pay attention?
Is there some sort of greybeard grapevine that old school webmasters can use to get the attention of class A IP network operators?
webmasters as in single operators of httpd somewhere, no, not likely
other network operators who are recognized ASNs and go to NANOG or RIPE or APNIC conferences and meet people face to face, yes.
if you wilfully disregard valid human-written abuse complaints coming from other equally-clueful network engineering persons at other ISPs, you and your ISP will acquire a tainted reputation.
there's various industry back-channels that exist between ISPs to communicate with each other about this sort of thing, I won't go into many details, primarily because there's a high degree of interest of the parties on all sides in keeping the email inboxes and phone numbers uncluttered with clueless single-residential-last-mile-user complaints about their wifi not working right.
"Hey Bob. Bunch of clients on your network keep initiating https connections with servers on my network"
"Gee Bob, sorry about that I'll go see what's causing that right away..."
If folks are escalating to the neteng crews it usually means something is broken via normal channels and it could use attention.
That's social network is exercised for a variety of things:
Abuse, ddos, IP spoofing, botnet C2s being hosted somewhere, malware hosting, complex customer investigations, route leaks, route hijacks, routing loops, congestion, executives home internet issues and more.
Feelings aside, it doesn't seem that different in principle to an ad blocker with a shared list of ad servers.
They sent back a template saying it's spoofed. As if I can't tell the difference between a From/Received header and the actual remote address that was delivering this garbage to my own mail server. All of this I already made clear in the original email.
Next up is Microsoft straight out rejecting email from my new IP address. It's not on any blacklist, it never did anything wrong, it just hasn't sent email before. Their systems for getting off the blacklist are somewhere between obscure, untouched for decades, broken, and nonexistent, depending on which page you look at and who you ask. Employees, if you get that far after hours of searching, claim my IP isn't blocked (yeah, right... they've blocked the range and so the employees can't find the IP ban, as if I'm supposed/able to know what bounds their range ban has.) The furthest I've got is that they somehow disable delivery failure notifications for a few weeks to shut me up but email still doesn't arrive.
I'm rather done dealing with Microsoft of late.
And then additional scoring based on DKIM fail as well.
GPT-[0-9]+, DALL-E, "AI".
ChatGPT: Produce a blog post in the style of [Given Name] [Surname]
From NCSA Mosaic 2.7, circa 1993, one of the first "web browsers", authored in part by one of Silicon Valley's favourite VC, the "a" in az16.com.
#
# Agent Spoofing -- Who will you be today?
#
# "I don't think [t]he[y]'ll be too keen" -- Originally from Monty Python, run
# into the ground by Tommy Reilly
#
# NOTE! There is a hard limit of 50 agents! Mosaic will not read anymore!
#
# This file should be located in your HOME directory under the filename:
# .mosaic-spoof-agents
#
# This file consists of lines that begin with a # (comment), a - (a seperator
# for the sub-menu off options), a + (this is special...it denotes that the
# agent following it [no spaces] should be the default agent to use) and
# anything else (considered an agent spoof line).
#
- This is a seperator...as long as the first thing is a dash
# This is a comment line
#
# This is the "example" agent. If you don't like it...comment it out.
# if you want it to be your default, put a + infront of it.
#
Bond_James_Bond/007 (BMW; Z3 Roadster) DOHC_inline_16-valve_4-cylinder
Gandalf/White (Shadowfax; Wind) White_Staff
Elric/Emperor_of_Melnibone (Hellsword; StormBringer - Devourer of Souls)
The sad reality is that "tech" companies and their service providers routinely use HTTP headers for their own manipulative, deceptive and user privacy-hostile commercial purposes. They use UA headers in ways that were not contemplated in 1993 and probably never intended. (Note I am not referring to spoofing, I am referring to data collection, tracking, e.g., "browser fingerprinting", and "surveillance capitalism" in general.)
Sending a UA header generally offers me, the end user, no benefits, so I do not send one.
Why not ban those Microsoft IPs or at least rate-limit them to a limited number of HTTP requests in a given time period.
Also looks like this website does not use sitemaps. At least, there is nothing listed in robots.txt. IMHO, there is no need to spider/crawl if there are sitemap XML files listing all the content URLs.
you never know if these are all URLs tho, so I would always crawl and brute force more paths than trusting sitemap.xml
UA spoofing is as old as 1994 or maybe 95 at the youngest.
There is a reason why every browser claims to be Mosaic Communicator.
Absolute magnitude is irrelevant. A standard deviation uptick vs baseline is always interesting.
This is just standard background noise.
There seems to be significant overlap between sysadmins who send handwritten abuse reports and “targeted individuals”/“gangstalking victims”.
Whichever way you cut that, if this happens to be a single host server, you're rapidly approaching a core-day with this load once you add some logging and such on top.
No way a scraper running at less than 1rps is anywhere near a core-day without some horrible misconfiguration, it should be way easier to address that than to fight a war against scrapers.
I have significantly more complicated flask sites handling 100+ rps on a single core, it doesn’t require any black magic.