I was wrong about robots.txt
evgeniipendragon.com
evgeniipendragon.com
robots.txt main purpose back in the day was curtailing penalties in the search engines when you got stuck maintaining a badly-built dynamic site that had tons of dynamic links and effectively got penalized for duplicate content. It was basically a way of saying "Hey search engines, these are the canonical URLs, ignore all the other ones with query parameters or whatever that give almost the same result."
It could also help keep 'nice' crawlers from getting stuck crawling an infinite number of pages on those sites.
Of course it never did anything for the 'bad' crawlers that would hammer your site! (And there were a lot of them, even back then.) That's what IP bans and such were for. You certainly wouldn't base it on something like User-Agent, which the user agent itself controlled! And you wouldn't expect the bad bots to play nicely just because you asked them.
That's about as naive as the Do-Not-Track header, which was basically kindly asking companies whose entire business is tracking people to just not do that thing that they got paid for.
Or the Evil Bit proposal, to suggest that malware should identify itself in the headers. "The Request for Comments recommended that the last remaining unused bit, the "Reserved Bit" in the IPv4 packet header, be used to indicate whether a packet had been sent with malicious intent, thus making computer security engineering an easy problem – simply ignore any messages with the evil bit set and trust the rest."
Or maybe more like the opposite: robots.txt told bots what not to touch, while sitemaps point them to what should be indexed. I didn’t realize its original purpose was to manage duplicate content penalties though. That adds a lot of historical context to how we think about SEO controls today.
That wasn’t its original purpose. It’s true that you didn’t want crawlers to read duplicate content, but it wasn’t because search engines penalised you for it – WWW search engines had only just been invented and they didn’t penalise duplicate content. It was mostly about stopping crawlers from unnecessarily consuming server resources. This is what the RFC from 1994 says:
> In 1993 and 1994 there have been occasions where robots have visited WWW servers where they weren't welcome for various reasons. Sometimes these reasons were robot specific, e.g. certain robots swamped servers with rapid-fire requests, or retrieved the same files repeatedly. In other situations robots traversed parts of WWW servers that weren't suitable, e.g. very deep virtual trees, duplicated information, temporary information, or cgi-scripts with side-effects (such as voting).
Very much so.
Computation was still expensive, and http servers were bad at running cgi scripts (particularly compared to the streamlined amazing things they can be today).
SEO considerations came way way later.
They were also used, and still are, by sites that have good reasons to not want results in search engines. Lots of court files and transcripts, for instance, are hidden behind robots.txt.
I think this is still relevant today in cases where there are not many resources available: think free tiers, smallest fixed cost/fixed allocation scenarios, etc.
It's the same as a noreply email, if you can get away with sticking your fingers in your ears and humming when someone is telling you something you don't want to hear, and you have a computer to hide behind, then it's all good.
I think you can reasonably assume that if a website went through the trouble of making such a modal (for legal compliance reasons), the functionality works (also for legal compliance reasons). And, you as the client can verify whether it works, and can choose not to store them regardless.
How do you do that? Cookies are typically opaque (encrypted or hashed) bags of bits.
Do-not-track is opt-out, not opt-in, and in fact, it is not opt-anything since browsers started to set it to "1" by default without asking. There is no law forcing advertisers to honor that.
I guess it could work the other way: if you set do-not-track to 0 (meaning "do-track"), which no browser does by default, make cookies auto-accept and do not show the banner. But then the law says that it should require no more actions to refuse consent than to consent (to counter those ridiculous "accept or uncheck 100 boxes" popups), so it would mean they would also have to honor do-not-track=1, which they don't want to.
I don't know how legislation could be unbroken. Users don't want ads, don't want tracking, they just want the service they ask for and don't want to pay for it. Service providers want exactly the opposite. Also people need services and services need users. There is no solution that will satisfy everyone.
Similarly consumer protection laws—which the cookie banners are—are not set to satisfy everyone, they are set such that companies cannot use their outsized power to exploit their customers. A good consumer protection law will simply ban harmful behavior regardless of whether companies which engage in said harmful behavior want are satisfied with that ban or not. A good consumer protection law, will satisfy the user (or rather the general public) but it may satisfy the companies.
Bad consumer protection laws try to pretend that trade offs don't exist. You don't want to see ads, that's fine, but now you either need to self-host that thing or pay someone else money to do it because they're no longer getting money from ads.
There is no point in having an opt in for tracking. If the user can be deprived of something for not opting in (i.e. you can't use the service) then it's useless, and if they can't then the number of people who would purposely opt in is entirely negligible and you ought to stop beating around the bush and do a tracking ban. But don't pretend that's not going to mean less "free stuff".
The problem is legislators are self-serving. They want to be seen doing something without actually forcing the trade off that would annihilate all of these companies, so instead they implement something compromised to claim they've done something even though they haven't actually done any good. Hence obnoxious cookie banners.
There are very many industries where this is exactly what happens. If you want a stack of lumber or a bag of oranges, it's a fungible commodity and there is no seller who can prevent you from buying the same thing from someone else if you don't like their terms.
If this is ever not the case, the thing you should be addressing is that, instead of trying to coerce an oligopoly that shouldn't exist into behaving under the threat of government penalties rather than competitive pressure. Because an uncompetitive market can screw you in ten thousand different ways regardless of whether you've made a dozen of them illegal.
> That means things like requiring sites to get permission to track, and punishing those that track users without permission. It's a good policy in theory, but it needs to be paired with good enforcement, and that's where things are currently lacking.
It's not a good policy in theory because the theory is ridiculous. If you have to consent to being tracked in exchange for nothing, nobody is going to do that. If you want a ban on tracking then call it what it is instead of trying to pretend that it isn't a ban on the "free services in exchange for tracking data" business model.
I actually agree with you that cookie banners are a bad policy, but for a different reason. As I understand it there are already requirements that the same service should also be available to opt-out users, however as your parent noted, enforcement is an issue. I, however, think that tracking users is extremely consumer hostile, and I think a much better policy would be a simple ban on targeted advertising.
There isn't an inherent power imbalance that exists between the company and their customers, when there is consumer choice. Which is why regulations that restrict rather than expand consumer choice are ill-conceived.
> If there is no way to doing a service for free without harming the customers, this service should be regulated such that no vendor is able to provide this service for free.
But that isn't what those regulations do, because legislators want to pretend to do something while not actually forcing the trade off inherent in really doing the thing they're only pretending to do.
> I, however, think that tracking users is extremely consumer hostile, and I think a much better policy would be a simple ban on targeted advertising.
Which is a misunderstanding of the problem.
What's actually happening in these markets is that we a) have laws that create a strong network effect (e.g. adversarial interoperability is constrained rather than required) which means that b) the largest networks win, and the networks available for free then becomes the largest.
Which in turn means you don't have a choice, because Facebook is tracking everyone but everybody else is using Facebook, which means you're stuck using Facebook.
If you ban the tracking while leaving Facebook as the incumbent, two things happen. First, those laws are extremely difficult to enforce because neither you nor the government can easily tell what they do with the information they inherently get from the use of a centralized service, so they aren't effective. And second, they come up with some other business model -- which will still be abusive because they still have market power from the network effect -- and then get to blame the new cash extraction scheme on the law.
Whereas if you do what you ought to do and facilitate adversarial interoperability, that still sinks their business model, because then people are accessing everything via user agents that block tracking and ads, but it does it while also breaking their network effect by opening up the networks so they can't use their market power to swap in some new abusive business model.
> There isn't an inherent power imbalance that exists between the company and their customers
That is very simplistic, and maybe idealistic from an unrealistic view of free-market capitalism. But there is certainly an inherent power imbalance. Before leaded gasoline was banned, it was extremely hard for environmentally conscious consumer to make the ethical choice and buy unleaded gasoline. Before seatbelts were required, a safety aware consumer might still have bought a car without one simply because the cars with seatbelts were either unavailable or unaffordable. Those aren’t real choices, but rather choices which are forced onto the consumer as a result of the competitive environment where the consumer hostile option generates much more revenue for the company.
The hard part isn't the rule, it's the enforcement.
To begin with, banning targeted advertising isn't really what you want to do anyway. If you have a sandwich shop in Pittsburgh and you put up billboards in Pittsburgh but not in Anchorage, you're targeting people in Pittsburgh. If you sell servers and you buy ads in a tech magazine, you're targeting tech people. I assume you're not proposing to require someone who wants to buy ads for their local independent pet store to have nearly all of them shown to people who are on the other side of the country?
What you're really trying to do is ban the use of individualized tracking data. But that's extremely difficult to detect, because if you tell Facebook "show this ad to people in Miami", how do you know if it's showing them to someone because they're viewing a post likely to be popular with people in Miami in general vs. because the company is keeping surveillance dossiers on every individual user?
The only thing that actually works is for them not to have the data to begin with. Which is the thing where you have to empower user agents to provably constrain what information services have about their users, i.e. adversarial interoperability.
> That is very simplistic, and maybe idealistic from an unrealistic view of free-market capitalism.
It's a factual description of competitive markets.
> Before leaded gasoline was banned, it was extremely hard for environmentally conscious consumer to make the ethical choice and buy unleaded gasoline.
The ban on leaded gasoline isn't a consumer protection regulation, it's an environmental regulation. Gas stations weren't selling leaded gasoline in spite of customers preferring unleaded, they were selling it because it was cheaper to make and therefore what customers preferred in the absence of a ban. It's a completely different category of problem and results from an externality in which the seller and the buyer both want the same thing but that thing harms some third party who isn't participating in the transaction.
> Before seatbelts were required, a safety aware consumer might still have bought a car without one simply because the cars with seatbelts were either unavailable or unaffordable.
This is how safety features evolve.
Seat belts were invented in the 19th century but we didn't start getting strong evidence of their effectiveness until the 1950s and 60s. Meanwhile that's the same period of time the US started building the interstate system with the corresponding increase in vehicle ownership, and therefore accidents.
So into the 1960s there was an increasing concern about vehicle safety, the percentage of cars offered with seat belts started increasing, and then Congress decided to mandate them -- which is what the market was already doing, because the customers (who are largely the same people as the voters) were demanding it.
That is a consistent trend. Things like that get mandated just as the majority of the market starts offering them, and then Congress swoops in to take credit for the benefit of what was already happening regardless.
What those laws really do is a) increase compliance costs (and therefore prices), and b) prohibit the minority of customers from buying something for specific reasons which is different than what the majority wants, because it's banned. For example, all cars are now required to have anti-lock brakes, but ABS can increase stopping distances on certain types of terrain. A professional driver who is buying a vehicle for specific use on those types of terrain is now prohibited from buying a vehicle without ABS on purpose even though it's known to cause safety problems for them.
> Those aren’t real choices, but rather choices which are forced onto the consumer as a result of the competitive environment where the consumer hostile option generates much more revenue for the company.
That type of choice is the thing that specifically doesn't happen in a competitive market, because then the consumer goes to a competitor.
Where it does happen is in uncompetitive markets, but in that case what you need is not to restrict the customer's choices, it's to increase competition.
IIRC IE10 did that, to much outcry because it upended the whole DNT being an explicit choice; no other browser (including Edge) set it as a default.
There have been thoughts about using DNT (the technical communication mechanism about consent/objection) in correlation with GDPR (the legal framework to enforce consent/objection compliance)
https://www.w3.org/blog/2018/do-not-track-and-the-gdpr/
The GDPR explicitly mentions objection via technical means:
> In the context of the use of information society services, and notwithstanding Directive 2002/58/EC, the data subject may exercise his or her right to object by automated means using technical specifications.
https://law.stackexchange.com/a/90002
People like to debate as to whether DNT itself has enough meaning:
> Due to the confusion about this header's meaning, it has effectively failed.
https://law.stackexchange.com/a/90004
I myself consider DNT as what it means at face value: I do not want to be tracked, by anyone, ever. I don't know what's "confusing" about that.
The only ones that are "confused" are the ones it would be detrimental to i.e the ones that perform and extract value from the tracking, and make people run in circles with contrived explanations.
It would be perfectly trivial for a browser to pop up a permission request per website like there is for webcams or microphone or notifications, and show no popup should I elect to blanket deny through global setting.
Unfortunately, the legal part of it failed, even in the EU.
Well, yes, the point is to tell the bots what you've decided to consider "bad" and will ban them for. So that they can avoid doing that.
Which of course only works to the degree that they're basically honest about who they are or at least incompetent at disguising themselves.
I always consider "good" a bot that doesn't disguise itself and follows the robots.txt rules. I may not consider good the final intent of the bot or the company behind it, but the crawler behaviour is fundamentally good.
Especially considering the fact that it is super easy to disguise a crawler and not follow the robots conventions
I admit I'm one of those people. After decades where I should perhaps be a bit more cynical, from time to time I am still shocked or saddened when I see people do things that benefit themselves over others.
But I kinda like having this attitude and expectation. Makes me feel healthier.
Trust by default, also by default, never ignoring suspicious signals.
Trust is not being naïve, I find the confusion of both very worrying.
Actually Veritasium has a great video about this. It's proven as the most effective strategy in monte carlo simulation.
EDIT: This one: https://youtu.be/mScpHTIi-kM
EDIT: i thought about it for a min, i think in the olden days a crawler crawling every path through a website could yield an inferior search index. So robots.txt gave search engines a hint on what content was valuable to index. The content provider gained because their SEO was better (and cpu util. lower) and the search engine gained because their index was better. So there was an advantage to cooperation then but with crawlers feeding LLMs that isn't the case.
Have you tried Anubis? It was all over the internet a few months ago. I wonder if it actually works well. https://github.com/TecharoHQ/anubis
On their README.md they state:
> This program is designed to help protect the small internet from the endless storm of requests that flood in from AI companies. Anubis is as lightweight as possible to ensure that everyone can afford to protect the communities closest to them.
I love the idea!
While I absolutely love the intent of this idea, it quickly falls apart when you're dealing with systems where you only get the signals after you've already lost everything of value.
There are varying degrees of this through our lives, where the trust lies not in the fact that people will just follow the rules because they are rules, but because the rules set expectations, allowing everyone to (more or less) know what's going on and decide accordingly. This also makes it easier to single out the people who do not think the rules apply to them so we can avoid trusting them (and, probably, avoid them in general).
Naturally I am talking about cultures where that decision has not been taken away from their citizens.
But yes, I do the same. I just do not come here to pretend this is virtue.
It should be a matter of judgement and not following rules just because.
It's not legal if somebody is following the slower car on the left and steers to the right to pass. However some drivers stick to the left at a speed slower than the limit and if they don't yield what happens is that eventually they get passed on the right.
The two cases have different names. The normal pass is "sorpasso", the other one (passing by not steering) is "superamento", which is odd but they had to find a word for it.
The problem with that is that most people are not educated enough to judge what makes sense and what doesn’t, and the less educated you are, the more likely you are to believe you know what makes sense when you’re actually wrong. These are exactly the people that should be following the rules blindly, until they actually put in the effort to learn why those rules exist.
It actually frightens me how true this statement is.
To reinforce my initial position about how important the rules are for setting expectations, I usually use cyclists as an example. Many follow the proposed rules, understanding they are traffic, and right of way is not automagically granted based on the choice of vehicle, having more to do with direction and the flow of said traffic.
But there's always a bad apple, a cyclist who assumes themselves to be exempt from the rules and rides against the flow of traffic, then wonders why they got clipped because a right-turning driver wasn't expecting a vehicle to be coming from the direction traffic is not supposed to come from.
In the end, it's not really about what we drive or how we get around, but whether we are self-aware enough to understand that the rules apply to us, and collectively so. Setting the expectation of what each of our behaviors will be is precisely what creates the safety that comes with following them, and only the dummies seem to be the ones who think they are exempt.
I suppose the same goes if you take the tube, ride a bike, walk, etc? There's still rules in terms of behavior, flow of traffic (even foot traffic), etc, that helps set a number of expectations so everyone can decide and behave accordingly. Happy to hear different thoughts on this!
That trust comes from the knowledge that it's likely that those drivers also don't want to crash, and would rather prefer to get where they're going.
It's usually a bad default to assume incompetence on the part of others, especially when many experienced and knowledgeable people have to be involved to make a thing happen.
The idea behind the DNT header was to back it up with legislation-- and sure you can't catch and prosecute all tracking, but there are limitations on the scale of criminal move fast and break things before someone rats you out. :P
The GDPR requires consent for tracking. DNT is a very clear "I do not consent" statement. It's a very widely known standard in the industry. It would therefore make sense that a court would eventually find companies not respecting it are in breach of the GDPR.
That was a theory at least...
https://en.wikipedia.org/wiki/Robots.txt
The scenario I remember was that the underfunded math department had an underpowered server connected via a wide and short pipe to the overfunded CS department and webcrawler experiments would crash the math department's web site repeatedly.
I'll bite. It seems like a poor strategy to trust by default.
That aggressive crawling to train those on everything is insane.
That kind of user experience is one that I've seen a lot on HN, and every time, without fail, it's because they're doing something that makes them look like a bot, and then being all Surprised Pikachu when they get treated like a bot by websites.
A random scraper, on the other hand, just racks up my AWS bill and contributes nothing in return. You'd have to be very, very convincing in your bot description (yes, I do check out the link in the user-agent string to see what the bot claims to be for) in order to justify using other people's resources on a large scale and not giving anything back.
An open web that is accessible to all sounds great, but that ideal only holds between consenting adults. Not parasites.
It really won’t. It will steal your website’s content and regurgitate it back out in a mangled form to any lazy prompt that gets prodded into it. GPT bots are a perfect example of the parasites you speak of that have destroyed any possibility of an open web.
I'll give them at most 3 years before sponsored links begin appearing in the output and "AI optimization" becomes a fashionable service alongside the SEO snake oil. Most publishers won't care whether their content is mangled or not, as long as it is regurgitated with the right keywords and links.
"Yes, there are a lot of great restaurants in Chicago that cater to vegans and people who enjoy musical theater. Dancing Dandelions in River North is one." or "One way to handle dogs defecating in your lawn is with Poop-Be-Gone, a non-toxic product that dissolves the poop."
It's not great for people who sell text online (journalists, I guess, who else?). But that's probably not the majority of content.
In other cases, like technical writing, you might want to protect the data. There is a danger that your content will be stolen and nothing will be given in return - traffic, money, references, etc.
If your webserver serves up the page, you've already pre-consented.
One of my retirement plans has a monthly statement available as a pdf document. We're allowed to download that. But the bot I wrote to download it once a month was having trouble, they used some fancy bot detection library to cockblock it. Wasn't allowed to use Mechanize. Why? Who the fuck knows. I'm only allowed to have that statement if I can be bothered to spend 15 minutes a month remembering how to fucking find it on their site and downloading it manually, rather than just saving a copy. Banks are even worse... they won't keep a copy of your statements longer than 6 months, but go apeshit if you try to have those automatically downloaded.
I don't ask permission or play nice anymore. Your robots.txt is ignorable, so I ignore it. I do what I want, and you're the problem not me.
It would be one thing if it were driving more users to my forum. But human usage hasn't changed much, and the bots drop cache hit rate from 70% to 4% because they go so deep into old forum content.
I'd be curious to see a breakdown of what the bots are doing. On demand searches? General data scraping? I ended up blocking them with CF's Bot Blocker toggle, but I'd allow them if it were doing something beneficial for me.
Part of the problem is all those companies who use AWS "standard practice" services, who assume the cost of bandwidth is just what AWS charges, and compute-per-page is just what it is, and don't even optimise those (e.g. S3/EC2/Lambda instead of CloudFront).
I've just compared AWS egress charge against the best I can trivially get at Hetzner (small cloud VMs for bulk serving https cache).
You get an astonishing 392x(!) more HTTPS egress from Hetzner for the same price, or equivalently 392x cheaper for the same amount.
You can comfortably serve 100+ TB/month that way. With 10,000 pages times 1000 bots per month, that gives you 10MB per page, which is more than almost any eCommerce site uses, when you factor that bots (other than very badly coded bots) won't fetch the common resources (JS etc.) repeatedly for each page, only the unique elements (e.g. HTML and per-product images).
10,000 is Monday. Morning.
My somewhat silly take on seeing a bunch of information like emails in a user agent string is that I don't want to know about your stupid bot. Just crawl my site with a normal user agent and if there's a problem I'll block you based on that problem. It's usually not a permanent block, and it's also usually setup with something like fail2ban so it's not usually an instant request drop. If you want to identify yourself as a bot, fine, but take a hint from googlebot and keep the user agent short with just your identifier and an optional short URL. Lots of bots respect this convention.
But I'm just now reminded of some "Palo Alto Networks" company that started dumping their garbage junk in my logs, they have the audacity to include messages in the user agent like "If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com" or "find out more about our scans in [link]". I put a rule in fail2ban to see if they'd take a hint (how about your dumb bot detects that it's blocked and stops/slows on its own accord?) but I forgot about it until now, seems they're still active. We'll see if they stop after being served nothing but zipbombs for a while before I just drop every request with that UA. It's not that I mind the scans, I'd just prefer to not even know they exist.
I also tend to see the web from the "open web" dream perspective. By default no traffic is blocked. The burden of requesting is already inherently done with a client -- they request a route, and I serve it or not. For things like my blog I don't tend to care who is requesting a particular route -- even admin pages can be requested, they just don't get anything without being logged in. If someone is being "cute" requesting non-existent wordpress pages or what have you, searching for vulnerabilities, or have an annoying/ugly user agent string, or are just pounding me for no real reason, then I do start to care. (The "pounding" aspect is a bit trickier -- I look at steady state. Another comment mentioned cutting their db server's cpu load in half by dropping unlikely-to-be-real-users from two countries. For me, if that is merely a steady state reduction from like 10% of a machine to 5%, I don't really care, I start caring when it would get in the way of real growth without having to use more resources.)
When I was hosting on EC2, I used to have very mild anxiety that I'd piss off someone and they'd try to "harm" me by launching a botnet of requests at large media files and rack up bandwidth costs. (I believe it when some people say this has happened more organically with normal bots in the age of LLMs, but my concern was more targeted botnets/ddos.) There are a few ways to mitigate that anxiety: 1) setup monitoring, alerts, and triggers directly in code running on the instance itself or via overseeing AWS tools (I did the latter, which is less reliable, but still. There was a threshold to shutdown the whole instance, minimizing the total damage possible to something like under a couple hundred bucks, I forget the details of trying to calculate how much traffic could theoretically be served before the monitoring side noticed) 2) hide behind cloudflare and their unlimited bandwidth, as my content was mostly static (I didn't do that) 3) move/rearchitect to a free host like github pages, give up hosting my own comments (again didn't do) 4) move to OVH which has unlimited bandwidth (did this when Amazon wanted to start charging an absurd amount for just a single ipv4 address).
I can see the argument that if I want to hide something, I should put it behind the layer of authentication. Robots is not a substitution for proper access control mechanisms. It is more of a "if they do honor this document, this would reduce the unnecessary traffic to my site" notion.
I appreciate you highlighting your personal experience in dealing with bots! I like the ideas of monitoring and being behind something like Cloudflare tools which would protect against the major influx of traffic. I think this is especially important for smaller sites which either use low or free tiers of cloud services.
Since it is impossible to know a priori which crawler are malicious, and many are malicious, it is reasonable to default to considering anything unknown malicious.
My foray into robots.txt started from the whole notion of AI companies training on everything they can put their hands on. I want to be able to have a say whether I allow it or not. While not all bots will honor the robots.txt file, there are plenty that do. One way that I found you can test that is by asking the model directly to scrape a particular link (assuming the model has browsing capabilities).
Bots are not malicious by default. It is what that company does with your data and how you feel about it that matters in the end.
It’s surprising to see the author frame what seems like a basic consequence of their actions as some kind of profound realization. I get that personal growth stories can be valuable, but this one reads more like a confession of obliviousness than a reflection with insight.
And then they posted it here for attention.
Objectively, "I give you one (1) URL and you traverse the link to it so you can get some metadata" still counts as crawling, but I think that's not how most people conceptualize the term.
It'd be like telling someone "I spent part of the last year travelling." and when they ask you where you went, you tell them you commuted to-and-fro your workplace five times a week. That's technically travelling, although the other person would naturally expect you to talk about a vacation or a work trip or something to that effect.
It’s definitely not crawling as robots.txt defines the term. :
> WWW Robots (also called wanderers or spiders) are programs that traverse many pages in the World Wide Web by recursively retrieving linked pages.
— https://www.robotstxt.org/orig.html
You will see that reflected in lots of software that respects robots.txt. For instance, if you fetch a URL with wget, then it won’t look at robots.txt. But if you mirror a site with wget, then it will fetch the initial URL, then it will find the links in that page, then before fetching subsequent pages it will fetch and check robots.txt.
Like other commenters have indicated, I will likely need to go back and allow some other social media to access the OPG data for previews to render properly. But since I mostly post on LinkedIn and HN, I don't feel the need to go and allow all of the other media at the moment. That might change in the future.
You block things -> of course good actors will respect and avoid you -> of course bad actors will just ignore it as it's a piece of "please do not do this" not a firewall blocking things.
I try to write about things that I learn or find interesting. Sharing it here in the hopes that others might find it interesting too.
The author had blocked all bots because they wanted to get rid of AI scrapers. Then they wanted to unblock bots scraping for OpenGraph embeds so they unblocked...LinkedIn specifically. What if I post a link to their post on Twitter or any of the many Mastodon instances? Now they'd have to manually unblock all of their UA, which they obviously won't, so this creates an even bigger power advantage to the big companies.
What we need is an ability to block "AI training" but allow "search indexing, opengraph, archival".
And of course, we'd need a legal framework to actually enforce this, but that's an entirely different can of worms.
Alternatively, it could make the request anyways and separate the crawled sites by permitted purpose in its output.
I do think that I will likely need to go back and unblock a couple of other bots for this exact reason - so that it would be possible to share it and have previews in other social media. I like to take a slow and thoughtful approach to allowing this traffic as I get to learn what it is that I want and do not want.
Comments here have been a great resource to learn more about this issue and see what other people value.
On the other end is user-requested URLs. Obviously a browser operated by a human shouldn't consider robots.txt. Almost as obviously, a tool subscribing to a specific iCal calendar feed shouldn't follow robots.txt because the human told it to access that URL. (I recall some service, can't remember if it was Google Calendar or Yahoo Pipes or something else that wouldn't let you subscribe to calendars blocked by robots.txt which seemed very wrong.)
The URL preview use case is somewhat murky. If the user is posting a single link and expecting it to generate a preview this very much isn't crawling. It is just fetching based on a specific user request. However if the user posts a long message with multiple links this is approaching crawling that message for links to discover. Overall I think this "URL preview on social media" probably shouldn't follow robots.txt but it isn't clear to me.
1)
You consider this about the Linkedin site but don't stop to think about other social networks. This is true about basically all of them. You may not post on Facebook, Bluesky, etc, but other people may like your links and post them there.
I recently ran into this as it turns out the Facebook entries in https://github.com/ai-robots-txt/ai.robots.txt also block the crawler FB uses for link previews.
2)
From your first post,
> But what about being closer to the top of the Google search results - you might ask? One, search engines crawling websites directly is only one variable in getting a higher search engine ranking. References from other websites will also factor into that.
Kinda .... it's technically true that you can rank in Google if you block them in robots.txt but it's going to take a lot more work. Also your listing will look worse (last time I saw this there was no site description, but that was a few years back). If you care about Google SEO traffic you maybe want to let them on your site.
1) I only considered LinkedIn alone since I have been posting there and here on HN, and that's it. I figured I will let it play out until I need to allow for more bots to access it. Your suggestion of other people wanting to share the links to the blog is a very valid one that I haven't thought about. I might need to allow several other platforms.
2) With Google and other search engines I have seen a trend towards the AI summaries. It seems like this is the new meta for search engines. And with that I believe it will reduce the organic traffic from those engines to the websites. So, I do not particularly feel that this is a loss for me.
I might eat my words in the future, but right now I think that social media and HN sharing is what will drive the most meaningful traffic to my blog. It is a word-of-mouth marketing, that I think is a lot more powerful than finding my blog in a Google search.
I will definitely need to go back and do some more research on this topic to make sure that I'm not shooting myself in the foot with these decisions. Comments here have been very helpful in considering other opinions and options.
You should also consider that a large proportion of search is purely navigational. What if someone is trying to find your blog and is typing 'evgenii pendragon'. AI summaries don't show for this kind of search.
Currently I can see your site is still indexed (disallowing Google won't remove a page from the index if it's already been indexed) and so you show in search results, but because you block Google in robots.txt it can't crawl content and so shows an awkward 'No information is available for this page.' message. If you continue to block Google in robots.txt eventually the links to your site will disappear entirely.
Even if you really don't want to let AI summarize your content, I would at least allow crawlers to your homepage.
I appreciate your suggestion!
Wikipedia has backups for this reason. AI companies ignore the readily available backups and instead crawl every page hundreds of times a day.
I think debian also recently spoke up about it.
Deny /honeypot in your robots.txt
Add <a href="/honeypot" style="display:none" aria-hidden="true">ban me</a> to your index.html
If an IP accesses that path, ban it.
Unrelated meta question but is the aria tag necessarily since display: none; should be removing the content from the flow?
The argument is basically to have them scrape your website indefinitely wasting their resources for the bots that decide to ignore your robots.txt (or any bot if you desire)
My goal with this post was to describe my personal experience with the problem, research, and the solution - the overall journey. I also wanted to write it in a manner that a non-technical person would be able to follow. Hence, being more explicit in my elaborations.
If you make a new page and never want it to enter the Google search index, adding it to robots.txt is fine, Google will never crawl it and it will never enter the index.
If you have an existing page that is currently indexed and want to remove it, adding that page to robots.txt is a BAD idea though. In the short term Google will continue to show the page in search results, but show it with no metadata (because it can't crawl it anymore). Even worse, Google won't notice up any noindex tags on the page, because robots.txt is blocking the page from being crawled!
Eventually Google will get the hint and remove the page from the index, but it can be a very frustrating time waiting for that to happen.
The only way to be sure that it will stay out of the results is to use a noindex tag. Which, as you mentioned, search engine bots need to "read" in the code. If the URL is blocked, the "noindex" cannot be read.
[1] https://developers.google.com/search/docs/crawling-indexing/... (refer to the red "Warning" section)
Yes, there's an ancient google reference parser in C++11 (which is undoubtedly handy for that one guy who is writing crawlers in C++), but not a lot for the much more prevalent Python and JavaScript crawler writers who just want to check if a path is ok or not.
Even if bot writers WANT to be good, it's much harder than it should be, particularly when lots of the robots info isn't even in the robots.txt files, it's in the index.html meta tags.
rel=nofollow is a bad name. It doesn’t actually forbid following the link and doesn’t serve the same purpose as robots.txt.
The problem it was trying to solve was that spammers would add links to their site anywhere that they could, and this would be treated by Google as the page the links were on endorsing the page they linked to as relevant content. rel=nofollow basically means “we do not endorse this link”. The specification makes this more clear:
> By adding rel="nofollow" to a hyperlink, a page indicates that the destination of that hyperlink should not be afforded any additional weight or ranking by user agents which perform link analysis upon web pages (e.g. search engines).
> nofollow is a bad name […] does not mean the same as robots exclusion standards
Shouldn't be that hard if someone WANT to be good.
There are good reasons why you’d not want to rely on clients providing this information when posting to LinkedIn (scams, phishing, etc); but it’s interesting to see an entirely different approach to the problem used here.
However I also understand that there are a number of reasons for a server to scrape the link. In no particular order:
1. scraping all the things, they might be useful sometimes in the future.
2. the page changes, goes 404, the client is reset and loses its db and can't rebuild the preview, but the client can rely on the server for that
3. it's faster for the client as the preview comes with the message and it does not have to issue some extra calls and parse a possibly slow page.
Anyway you write that's the sender that generates the preview on iMessages so that's leaves point #1 and possibly the part of #2 about flaky internet connections: the server is in a better place to keep trying to generate the preview.
The result will be that giant corporations and those will bad intents will still find a way to access what they need, but small, hobby citizen and civil society efforts will be blocked out.
I very much dislike the invasive scrapping approaches. If something were to be done about it, it would result in a new way that clients interact with resources on the web.
If you don't want people to crawl your content, don't put it online.
There are so many consequences of disallowing robots -- what about the Internet Archive for example?
I want to post online but I don't want random asshole driveby's involved.
I sometimes put things online for specific people to view/use, or for my own purposes. That gets an “all crawlers can do one” robots.txt and sometimes a script or two to try waste a little the time of those that don't play ball.
It is online because I want it online, not for some random to hoover up.
I consider robots.txt as a garden gate. I know it isn't secure, but likewise someone peering directly into my back bedroom window knows just as well that I don't want them there.
I could put stuff like that behind authentication, but that is hassle for both me and the people who I want to be accessing the stuff. I usually use not-randomly-guessable URIs though sometimes that is inconvenient too, and anyway they do sometimes get “guessed”. I must have at least one friend-of-friend who has an active infestation which is reading their messages or browser history for things to probe because the traffic pattern isn't just preview generation, I've had known AI crawlers pass by some things repeatedly.
TBH I don't really care that much, much at all in fact, I just don't like the presumption that my stuff is free for commercial exploitation.
A human being who stops by to spend 20 minutes reading your blog once won't even notice they've spent 1.2 cents leafing through. This technology has existed for a while, and yet very few people have found it a good idea to wrap around. There is probably a good reason for that.
The realistic solution is to probably just do some statistics and figure out who's getting on your nerves, and then ban them from your digital abode. Annoying, but people go a lot farther to protect their actual homes if they happen to live in high crime areas.
While I doubt it does much today, that file really only matters to those that want to play by the rules which on the free web is not an awful lot of the web anymore I’m afraid.
> These rules are not a form of access authorization.
Meaning that these are not enforced in any way. They cannot prevent you from accessing anything really.
I think the only approach that could work in this scenario would be to find which companies disregard the robots.txt, and bring it to the attention of technical community. Practices like these could make the company look shady and untrustworthy if found out. That could be one way to keep them accountable, even though there is still no guarantee they will abide by it.
Yet, this post of his (posting his own work) gained traction. I believe for robots.txt topic rather than the article itself.
That shows that even if you ignore all the rules of keeping a healthy community (not publish your self promotion only), eventually you’d get traction and nobody would care, I guess.
Quick edit: my bad, wrong click brought me to the wrong location. So I made a bit wrong assumption. The author posted 4 extra posts alongside his own, so it’s not 100% of self-promotion, but 50%.
There are more articles available in my blog than I have shared here. I don't think that everything that I write is shareworthy on HN. There are some that I find to be more interesting. Those are the ones I end up sharing.
Like you have noticed, I try to share other interesting resources that I find online too. Is there a ratio of self/others content that would help keeping a healthy community?
If it was some other website, I’d say that if you post many of your own posts yourself many community members won’t be happy. I’d say go with not 50% but, say, at least 20% or even 10%. Again, my personal opinion of this website, it’s long beyond repair (if it ever was), so feel free to do whatever you want, unless some whims of mods ban you for no real reason one day.
I will consider to bring in more content that is other than my own. I did read the guidelines for HN, and I saw that it encourages members to share what they find, and occasionally their own stuff. Appreciate your opinion!
I wish there were way less posts like this.
I get not wanting to have our data serve as training data, but I've also seen moderately large newspapers throwing literally all LLM bots in there, i.e. not only those that scrape training data, but also those that process users' search requests or even direct article fetches.
The obvious, but possibly not expected, result was that this newspaper became effectively invisible to user searches in ChatGPT. Not sure if I'm an outlier here, but I personally click through to many of the sources ChatGPT provides, so they must be losing tons of traffic that way.
As far as I remember from google search console, a disallow directive in robots.txt causes google not only to avoid crawling the page, but also to eventually remove the page from its index. It certainly shouldn't add any more pages to its index, external references or not.
It is fun to learn something when you discover an unintended consequence and then work backwards from it.
You aren't going to get advertising without also providing value - be that money or information. Google has over 2 trillion in capitalization based primarily on the idea of charging people to get additional exposure, beyond what the information on their site otherwise would get.
My approach right now is to rely on social media traffic primarily where you can engage with the readers and build trust with the audience. I don't plan on using any advertising in the near future. While that might change, I am convinced that more intentional referral traffic will generate more intentional engagement.
It is amazing what people think is important these days.
Fortunately, the post inspector helps you suss out what's missing in some cases, but c'mon, man, how much effort should I spend helping a social media site figure out how to render a preview? Once you get it right, and to quote my 13 year old: "We have arrived, father... but at what cost?"
I didn't know that about the doctype tags. I must have had them right from the beginning. Didn't encounter those issues. It good to know though.
Note that in 90% of the cases this was unitentional. Just Cloudflare enabeling lazy contracters with overzealous defaults. And as I said before, yes, you can write to get this corrected, and maybe in the next maitainance update next quarter it might get fixed. Until the next cheapest contracter wins the bid, wipes the config to minimize mai tainance, and upu can startall over again.
The temaing 10% in case you wonder are malicious public bodies that want to pretend they do comply in making data public, but will do anything to make it as hard as possible to actually get to or consume yhat data. They love not just these blocking shenanigans, but will also hinder human browsing by putting in no translate pragmas, UX anti patterns, publish as poorly scanned PDFs, do not maintain canonical urls and shift things round constantly, remove things entirely "by mistake" even within the statutory period etc.
Then there's very rarely (>.5%) activists IT employees on a true 'anti AI' crusade. This is where you encounter the most ridiculous homebrew 'solutions' that will often result in hurting also normal people accessing the site. This to them is just collateral in their rightous war.
It used to be this ultrafake eternal job interview site, but people now seem uninhibited to go on wild political rants even there.