AI crawlers need to be more respectful
about.readthedocs.com
about.readthedocs.com
They say a single crawler downloaded 73TB of zipped HTML files in a month. That averages out to ~29 MB/s of traffic, every second, for an entire month.
Averaging 30 megabytes a second of traffic for a month is crossing into reckless territory. I don't think any sane engineer would call that normal or healthy for scraping a site like ReadTheDocs; Twitter/Facebook/LinkedIn/etc, sure, but not ReadTheDocs.
To me, this crosses into "recklessly negligent" territory, and I think should come with government fines for the company that did it. Scraping is totally fine to me, but it needs to be done either a) at a pace that will not impact the provider (read: slowly), or b) with some kind of prior agreement that the provider is accepting responsibility to provide enough capacity.
While I agree that putting content out into the public means it can be scraped, I don't think that necessarily implies scrapers can do their thing at whatever rate they want. As a provider, there's very little difference to me between getting DDoSed and getting scraped to death; both ruin the experience for users.
Wow. That's some seriously disrespectful crawling.
If I would design a crawler, I'd keep at least some basic form of tracking, if only to check for people deliberately trolling me by delivering me an infinite chain of garbage.
Naming and shaming the company while you're trying to work with them is a real good way to not get what you want.
Meta/Google/Whoever may benefit from economies of scale, so they're not seeing the full $5,000 their side, but they're hitting tens of thousands of sites with that crawler.
They are a persistent source of spam email servers, scrapers, and bot probes.
The simple reason is the operators quickly dump a host, and the next user is left wondering why their legitimate site is instantly spelunking spam ban lists.
It is the degenerative nature of cloud services... and unsurprisingly we end up often banning most parts of Digital Ocean, Amazon, Azure, Google, and Baidu.
Have a wonderful day, =3
We also follow the tit-for-tat forgiveness policy to ensure old bans are given a second chance. Mostly, we want the nuisance to sometimes randomly work, as it wastes more of their time fixing bugs.
And note, if a server is compromised and persistently causing a problem... we won't hesitate to black hole an entire country along with the active Tor exit nodes and known proxies lists (the hidden feature in context cookies).
Have a great day friend, =3
Don't worry about it friend =3
They may visit again, but are less likely to mess with the platform. Note, cons never buy anything... ever... it is against their temperament.
I find it interesting several of cons on YC are upset by someone else's administrative policies. Reddit should enforce these policies too, or at least drop a country or pirate flag icon beside nasty posts...
Have a great day, and don't fear the ban hammer friend =3
1. masquerading as YC associated organizations
2. attempting to extract banking information with classic 419'er tactics
3. spamming member contact routes with several phone and email Grifts
These folks appear to be operating on a 4 month cycle, and redistributing target leads to other fraudsters.
Annoying, but hardly a problem limited to YC if that is your concern.
Best of luck, =3
If they keep hitting the limit within an hour 4+ times, than get fail2ban to block the IP for 2 days.
73TB is a fair amount to have on a cloud... usually at >30TiB firms must diversify with un-metered server racks, and CDN providers (traditional for large media files etc.)
Good luck =3
There is page referral monitoring, context cookies, and dynamically created link chaff with depth-charge rules.
One can dance all day friend, but a black-hole is coming for the entire IP block shortly. And unlike many people, some never remove a providers leased block and route until it is re-sold. =3
Domestic "Users" functioning as proxies will be tripping usage limits, and getting temporarily banned. Google does this by the way, try hammering their services and find out what happens.
Context cookies also immediately flag egregious multi-user routes, and if it is a ISP IP you know its a problem user. If it is over 15 users an hour per IP, than you can be 100% sure its a Tor proxy.
we ban over 243000+ IPs, and saw zero impact to our bottom line.
Have a nice day, =)
And again, none of this is simple. It has taken me a few weeks to establish a usage mechanism that does catch the worst feed pullers, but it still can hurt legit new users. That is an opportunity cost.
Allowing users known to have an active RAT or their "proxy friends" on a commercial site is not helping anyone.... especially the victims.
https://www.youtube.com/watch?v=aCbfMkh940Q
Worth studying the problem from time to time when you get bored of the antics.
These folks are generally uninterested in positively contributing to any community, but rather show up to cause trouble for fun and profit.
User API quotas are popular for a reason. =3
Have a nice day, =3
Don't take this trend personally friend, as if we see a fake iPhone sporting bandwidth >1300Mbps+... than the host is getting permanently banned anyway.
Have a wonderful day, =3
Click around a few times on any of your sites and looks like I'll be banned?
Multi tasking? Opening multiple interesting links?
Like what.
1. gets incrementally slower until firewall user rate limiting tokens refill the bucket (chokes >6MiB/min bandwidth use, and enforces abnormal traffic ban rules.)
2. Pauses serving a page if you spider though 6+ pages a minute (chokes speculative downloading)
3. if you violate site usage rules 4+ times in the past hour, than your get a 2 day IP ban
4. if you trip a spider trap, than you get a 5 day ban
5. If you are issued more than 5 context cookies, than the IP will get spammed with a captcha on every page for 5 days
6. If you violate any number of additional signatures (shodan etc.) than you get your IP block and route permanently banned. There is only 1 exception to this rule, and we don't share that with anyone.
7. The site content navigation is programmatically generated in JavaScript
8. The legal notice is very real for some people
Have a nice day friend, =)
Don't worry about it friend =3
From the article:
"This was a bug in their crawler that was causing it to download the same files over and over again."
People hitting the same files again sounds like a developer testing their code.
The crazy thing here is that the target site is content to pay ~$700 for the volume of traffic that you can move through a single teeny-tiny included-at-no-extra-charge cat5 Ethernet link in one single day. And apparently, they're going to continue doing so.
Over 99% of the bandwidth (and CPU) taken by the biggest podcast / music services simply on polling feeds is completely unnecessary. But ofc pointing this out to them gets some sort of "oh this is normal, we don't care" response because they are big enough to know that eg podcasters need them.
If you look at the "Defences" section: https://www.earth.org.uk/RSS-efficiency.html#Hints you'll see there are some things that can be done, such as randomly rejecting a large fraction of requests that don't allow compression (gzip is madly effective on many feed files: it's rude for a client not to allow it). But all these measures take effort to set up, and don't stop the bad bots making the request 100s of times too often. Just responding to each stupid request forces a flurry of packets and wakes up and uses CPU...
I don't disagree with your post. But: RSS downloads are at an all time low, and that's a bad thing.
They're at an all time low because Spotify and Apple both fetch feeds from centralized servers. 1000 subscribers no longer means 24000ish daily feed fetches, it means 48. With keep alive or H2, these services simply don't reconnect. The number of IPs that hit me from Apple, for instance, is probably only double digits.
Since Apple and Spotify both sit between me and the listeners, they eliminate the privacy that listeners would otherwise enjoy. It also forces podcasters to go to them to find out how many people are subscribed, which means lots of big databases instead of one database that I host for my customers.
Centralization of feed checking carries huge risks, in my opinion, especially as both Apple and Spotify make moves to also become the hosting providers.
Also I agree that the re-centralisation is a bad thing, mainly.
(I'd like to move to email to discuss this further, if possible: I have an arXiv paper to write!)
> Clients (hopefully bots) that disregard robots.txt and connect to your instance of HellPot will suffer eternal consequences. HellPot will send an infinite stream of data that is just close enough to being a real website that they might just stick around until their soul is ripped apart and they cease to exist.
As someone who written a lot of crawling infrastructure and managed large scale crawling operations, respectful crawling is important.
That being said it always seems like google has had a massively unfair advantage for crawling not only with budget but with brandname, and perceived value. It sometimes felt hard to reach out to websites and ask them to allow our crawlers, and grey tactics were often used. And I'm always for a more open internet.
I think regular releases of content in a compressed format would go a long way, but there would always be a race for having the freshest content. What might be better is offering the content in machine format, XML or JSON or even SOAP. Which is usually better for what the sites crawling want to achieve, cheaper for you to serve, and cheaper and less resource intensive compared to crawling. (Have them "cache" locally by enforcing rate limiting and signup)
VCs and other startup culture evangelists are always challenging founders to figure out what their ‘unfair advantage’ is.
That’s the name of the game.
1. I intentionally made sure my crawler was slow (I prefer batch processing workflows in general, and this also has the effect of not needing a machine gun crawler rate)
2. For data updates, I made sure to first do a HEAD request and only access the page if it has actually been changed. This is good for me (lower cost), the site owner, and the internet as a whole (minimizes redundant data transfer volume)
Regarding individual site policies, I feel there’s often a “tragedy of the commons” dilemma for any market segment subject to aggregator dominance:
- individual sites often aggressively hide things like pricing information and explicitly disallow crawlers from accessing them
- humans end up having to access them: this results in a given site either not being included at all, or accessed once but never reaccessed, causing aggregator data to go stale
- aggregators often outrank individual sites due to better SEO and likely human preference of aggregators, because it saves them research time
- this results in the original site being put at a competitive disadvantage in SEO, since the their product ends up not being listed, or listed with outdated/incorrect information
- that sequence of events leads to negative business outcomes, especially for smaller businesses who often already have a higher chance of failure
Therefore, I believe it’s important to have some sort of standard policy that is implemented and enforced at various levels: CDNs, ISPs, etc.
The policy should be carefully balanced to consider all these factors as well as having a baked in mechanism for low friction amendment based on future emergent effects.
This would result in a much better internet, one that has the property of GINI regulation, ensuring well-distributed outcomes that are optimized for global socioeconomic prosperity as a whole.
Curious to hear others’ perspectives about this idea and how one would even kick off such an ambitious effort.
> We have IP-based rate limiting in place for many of our endpoints, however these crawlers are coming from a large number of IP addresses, so our rate limiting is not effective.
Do you have something else in mind? Just shut down the whole site after a certain limit?
Just 3 AI spiders put more load on our servers than all search engine spiders and all human traffic combined.
Some numbers I have handy from before I blocked the bots:
ClaudeBot drove more requests through our Redmine in a month than it saw in the combined 5 years prior to ClaudeBot.
Bytespider accounted for 59% of the total traffic to our Git server.
Amazonbot accounted for 21% of the total traffic to our Git server.
Google has never even been close to breaking out of the single-digit-percentages of any metric.
But these days where they just rip content from the site to give people as answers, completely depriving the site of traffic, yeah that seems basically just as bad as the AI bots.
HN will cheer on a lot of things that are counter-intuitive to their wellbeing; open-weight models doesn't feel like one of them. You can't protest AI (or search engines) because after long enough people can't do their job without them. The correct course of action is to name-and-shame, not write pithy engineering blogs begging people to stop. People won't stop.
Except for the fact that they come from undisclosed sources from a company that does this: https://x.com/Tantacrul/status/1794863603964891567
You post it, others consume it. Same as it ever was.
As Squidward says: nobody gives a care for the fate of labor as long as they get their instant gratifications.
Invoice the abusers.
They're rolling in investor hype money, and they're obviously not spending it on competent developers if their bots behave like this, so there should be plenty left to cover costs.
You would be fooling yourselves if you think such a firm cared about robots.txt or page tags.
We warned them they would be sued eventually, to contact the site owners for legal access to the data, and issued a hard pass on the project. Probably they assumed if the indexing process was out of another jurisdiction their domestic firm wouldn't be liable for theft of service or copyright infringement.
It was my understanding AI/ML does not change legal obligations in business, but the firm probably found someone to build that dubious project eventually...
Spider traps and rate-limiting are good options too. =3
Obviously “anything goes” in civil suits however - if someone is being absurdly egregious with their crawling there’s usually some exposure to one tort or another.
And Reddit has definitely become more proactive about scrapers. ;-)
One may be sued, but not because you parsed robots.txt wrong =3
My understanding is that this is not accurate.
HiQ v LinkedIn established that this is only the case if you actually agreed to the terms of service. Such "agreement" only happens if the information is walled behind an account creation process, e.g. Facebook, Inc. v. Power Ventures, Inc. If it's just scraping publicly available webpages, the only legal issue with scraping would be unreasonably or obviously negligent scraping practices which lead to degradation or denial-of-service. And obviously the line for that would have to be determined in civil court.
eBay v. Bidder's Edge (2000) is the last case that I could find which even considered violation of robots.txt as very minor part of the judgement, but the findings were based far more on other things. Intel Corp. v. Hamidi also implicitly overruled the judgement in that that ruling (though not related to robots.txt, which was really just a very minor point in the first place).
One thing is for certain, is its jurisdictional... and way too messy to be responsible for maintaining/hosting (the ambiguous copyright protection outside a research context looked way too risky.) =3
This doesn't apply if you don't ever agree to anything - which is the case if the information is not locked behind account creation.
However, if they scraped the content using these account credentials, than it becomes a problem in a commercial context.
If I recall, only journalists and academics could argue Fair use at that point.
Anyway, I didn't touch the project mostly for copyright and trademark risk concerns.
Have a great day =3
OpenAI introduced gptbot in August 2023… they already took everything
In this case, they showed up to the data buffet long after it went rotten due to SEO.
Have a nice day =3
Hoping they just stop seems futile…
The only web crawler that did anything for me was Google, as Google sent an appreciable amount of traffic. Referrers from Bing were almost undetectable: the joke among my black hat SEO friends at the time was that you could rank for money keywords like "buy wow gold" and get 10 hits. Then there were the Chinese crawlers like Baidu that would crawl at 10x the rate of Google but send zero referrers. And then there were crawlers looking for copyrighted images that cost me money to accommodate even if they never sent me cease and desist letters.
As much as I hate the Google monopoly I couldn't afford having my site crawled like that without any benefit to me.
It's an awful situation for the long term though because it prevents new entrants. Right now I am thinking about a new search engine for a vertical where a huge number of products are available from different vendors and when you do find results from Google they are sold out at least 70% of the time. I hate to think it's going to get harder to make something.