A random scraper, on the other hand, just racks up my AWS bill and contributes nothing in return. You'd have to be very, very convincing in your bot description (yes, I do check out the link in the user-agent string to see what the bot claims to be for) in order to justify using other people's resources on a large scale and not giving anything back.
An open web that is accessible to all sounds great, but that ideal only holds between consenting adults. Not parasites.
It really won’t. It will steal your website’s content and regurgitate it back out in a mangled form to any lazy prompt that gets prodded into it. GPT bots are a perfect example of the parasites you speak of that have destroyed any possibility of an open web.
I'll give them at most 3 years before sponsored links begin appearing in the output and "AI optimization" becomes a fashionable service alongside the SEO snake oil. Most publishers won't care whether their content is mangled or not, as long as it is regurgitated with the right keywords and links.
"Yes, there are a lot of great restaurants in Chicago that cater to vegans and people who enjoy musical theater. Dancing Dandelions in River North is one." or "One way to handle dogs defecating in your lawn is with Poop-Be-Gone, a non-toxic product that dissolves the poop."
It's not great for people who sell text online (journalists, I guess, who else?). But that's probably not the majority of content.
In other cases, like technical writing, you might want to protect the data. There is a danger that your content will be stolen and nothing will be given in return - traffic, money, references, etc.
If your webserver serves up the page, you've already pre-consented.
One of my retirement plans has a monthly statement available as a pdf document. We're allowed to download that. But the bot I wrote to download it once a month was having trouble, they used some fancy bot detection library to cockblock it. Wasn't allowed to use Mechanize. Why? Who the fuck knows. I'm only allowed to have that statement if I can be bothered to spend 15 minutes a month remembering how to fucking find it on their site and downloading it manually, rather than just saving a copy. Banks are even worse... they won't keep a copy of your statements longer than 6 months, but go apeshit if you try to have those automatically downloaded.
I don't ask permission or play nice anymore. Your robots.txt is ignorable, so I ignore it. I do what I want, and you're the problem not me.
It would be one thing if it were driving more users to my forum. But human usage hasn't changed much, and the bots drop cache hit rate from 70% to 4% because they go so deep into old forum content.
I'd be curious to see a breakdown of what the bots are doing. On demand searches? General data scraping? I ended up blocking them with CF's Bot Blocker toggle, but I'd allow them if it were doing something beneficial for me.
Part of the problem is all those companies who use AWS "standard practice" services, who assume the cost of bandwidth is just what AWS charges, and compute-per-page is just what it is, and don't even optimise those (e.g. S3/EC2/Lambda instead of CloudFront).
I've just compared AWS egress charge against the best I can trivially get at Hetzner (small cloud VMs for bulk serving https cache).
You get an astonishing 392x(!) more HTTPS egress from Hetzner for the same price, or equivalently 392x cheaper for the same amount.
You can comfortably serve 100+ TB/month that way. With 10,000 pages times 1000 bots per month, that gives you 10MB per page, which is more than almost any eCommerce site uses, when you factor that bots (other than very badly coded bots) won't fetch the common resources (JS etc.) repeatedly for each page, only the unique elements (e.g. HTML and per-product images).
10,000 is Monday. Morning.
My foray into robots.txt started from the whole notion of AI companies training on everything they can put their hands on. I want to be able to have a say whether I allow it or not. While not all bots will honor the robots.txt file, there are plenty that do. One way that I found you can test that is by asking the model directly to scrape a particular link (assuming the model has browsing capabilities).
Bots are not malicious by default. It is what that company does with your data and how you feel about it that matters in the end.
I'll bite. It seems like a poor strategy to trust by default.
That aggressive crawling to train those on everything is insane.
That kind of user experience is one that I've seen a lot on HN, and every time, without fail, it's because they're doing something that makes them look like a bot, and then being all Surprised Pikachu when they get treated like a bot by websites.
Since it is impossible to know a priori which crawler are malicious, and many are malicious, it is reasonable to default to considering anything unknown malicious.
My somewhat silly take on seeing a bunch of information like emails in a user agent string is that I don't want to know about your stupid bot. Just crawl my site with a normal user agent and if there's a problem I'll block you based on that problem. It's usually not a permanent block, and it's also usually setup with something like fail2ban so it's not usually an instant request drop. If you want to identify yourself as a bot, fine, but take a hint from googlebot and keep the user agent short with just your identifier and an optional short URL. Lots of bots respect this convention.
But I'm just now reminded of some "Palo Alto Networks" company that started dumping their garbage junk in my logs, they have the audacity to include messages in the user agent like "If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com" or "find out more about our scans in [link]". I put a rule in fail2ban to see if they'd take a hint (how about your dumb bot detects that it's blocked and stops/slows on its own accord?) but I forgot about it until now, seems they're still active. We'll see if they stop after being served nothing but zipbombs for a while before I just drop every request with that UA. It's not that I mind the scans, I'd just prefer to not even know they exist.
I also tend to see the web from the "open web" dream perspective. By default no traffic is blocked. The burden of requesting is already inherently done with a client -- they request a route, and I serve it or not. For things like my blog I don't tend to care who is requesting a particular route -- even admin pages can be requested, they just don't get anything without being logged in. If someone is being "cute" requesting non-existent wordpress pages or what have you, searching for vulnerabilities, or have an annoying/ugly user agent string, or are just pounding me for no real reason, then I do start to care. (The "pounding" aspect is a bit trickier -- I look at steady state. Another comment mentioned cutting their db server's cpu load in half by dropping unlikely-to-be-real-users from two countries. For me, if that is merely a steady state reduction from like 10% of a machine to 5%, I don't really care, I start caring when it would get in the way of real growth without having to use more resources.)
When I was hosting on EC2, I used to have very mild anxiety that I'd piss off someone and they'd try to "harm" me by launching a botnet of requests at large media files and rack up bandwidth costs. (I believe it when some people say this has happened more organically with normal bots in the age of LLMs, but my concern was more targeted botnets/ddos.) There are a few ways to mitigate that anxiety: 1) setup monitoring, alerts, and triggers directly in code running on the instance itself or via overseeing AWS tools (I did the latter, which is less reliable, but still. There was a threshold to shutdown the whole instance, minimizing the total damage possible to something like under a couple hundred bucks, I forget the details of trying to calculate how much traffic could theoretically be served before the monitoring side noticed) 2) hide behind cloudflare and their unlimited bandwidth, as my content was mostly static (I didn't do that) 3) move/rearchitect to a free host like github pages, give up hosting my own comments (again didn't do) 4) move to OVH which has unlimited bandwidth (did this when Amazon wanted to start charging an absurd amount for just a single ipv4 address).
I can see the argument that if I want to hide something, I should put it behind the layer of authentication. Robots is not a substitution for proper access control mechanisms. It is more of a "if they do honor this document, this would reduce the unnecessary traffic to my site" notion.
I appreciate you highlighting your personal experience in dealing with bots! I like the ideas of monitoring and being behind something like Cloudflare tools which would protect against the major influx of traffic. I think this is especially important for smaller sites which either use low or free tiers of cloud services.