Hacker News Reader

Comment by smosher | Hacker News Reader

The rationale is weak. Some data is simply not worth indexing, and not worth serving up to bots. The flipside is: your crawler doesn't need to fetch everything on my site, and I'd be happy to ban all non-conforming bots site-wide.

It's not just about the functionality, but also a show of good faith and basic respect. If you're a bot author who knowingly violates my site policy I'd rather you didn't communicate with my web server at all.

robots.txt isn't perfect. Ideally a web server would be configured to deny bots access to restricted content via some sort of dnsbl mechanism (or CPAN/whatever module.) Or do both and ban the non-conforming site-wide.

The above notwithstanding, I'm voting for this article. It doesn't betray the usual cowardice by hiding the assertion behind the presumptuous Why.

View on HN

If you're a bot author who knowingly violates my site policy I'd rather you didn't communicate with my web server at all.

Quite. My robots.txt reads:

    User-agent: *
    Disallow: /cgi-bin/
    Disallow: /scratch/

I do not believe, as the archiveteam.org seem to think I do, that I have either stopped bots interfering with CGI scripts or spidering semi-private, temporary content. Indeed, my server logs show clearly enough that bots do these things all the time. Instead, the bot has violated my site's policy and its IP address will be considered for an iptables ban. It is a matter of which bots I consider fair game for rough treatment.

This is a common attitude to robots.txt, I think, and it does not reflect well on archiveteam.org's competence that they seem to be unaware of it.

Here's a concrete example of a site not worth archiving - back in the early days of the web (1994?) a friend of mine wrote a 4 function calculator. Each button was a hyperlink, so "34-12" meant going to the pages "?", "?3", "?34", "?34-", "?34-1", and "?34-12". That's an infinite number of web pages which should not be indexed.

I remember some anti-spam vigilante code that told legitimate bots to go away with robots.txt, told humans what the game was about, and attacked e-mail address scrapers by presenting an infinite number of fake e-mail addresses. "Click here for 1000 more e-mail addresses that will never bother a human!".

Not all of them were fake. Some of them were spam traps which never appeared anywhere else. Some of those spam traps were also coded so you could track down when they were served to connect the spiders to the spewers.

At least, that's the way I used to roll.

I love it!

Ah very good! I was trying to think of a short example that wasn't contrived with this very behaviour.

"Some data is simply not worth indexing, and not worth serving up to bots."

The broader points being made in the article are that the value of information is determined by the visitor, and that the burden of keeping the site up and running should fall on the host.

By that reasoning, the presence of a ROBOTS.TXT signifies "damage" or "temporary madness", and should therefore be ignored.

Also, keep in mind where this article came from - the Archive Team are bloody-minded about preserving information.

OK, wait.

If the data has value to the user, shouldn't the user be paying the host for the cost of making that data available (or perhaps considerably more, if it has a lot of value to the user)?

Excellent. How much are you paying ycombinator for your account?

If the answer is "nothing", then I guess you've just told me that all your comments are useless, and I should ignore them.

I'm committing the classic mistake of talking about the karma system here, but what, precisely, is wrong with my statement?

Drivingmenuts says the user should pay for hosting data. HN is hosting his data, but he doesn't pay for it. Is there some error in my logic, here, some flaw in my conclusion?

the Archive Team are bloody-minded about preserving information

They won't do a good job of that if they encourage too many people to make IP blocks of their machines.

AT, the organization, doesn't own any machines. Operations are all done by members, the majority of whom use consumer ISPs.

Do all of them ignore robots.txt?

Well, when I iterated through the everything2.com namespace and downloaded the majority of their content[1], I respected their robots.txt. After I did that, they changed it to:

  User-agent: * 
  Disallow: /

Does that count?

1: http://bbot.org/blog/archives/2011/01/17/more_fun_with_wget/

the value of information is determined by the visitor

That may be so in a sense, unless the visitor has no means of making the call -- or does so poorly. I promise you that every robots.txt ignoring bot makes terrible judgments in that regard. (I'm not trying to overstate anything here; those that abide it make, on average, slightly less terrible judgments.)

But it's not just the visitor who gets to make judgment calls. The value of serving those pages is something the host can decide. Belligerent drunks who abuse the staff aren't allowed in the coffee shop, and if they do it often enough they're not allowed back in when they sober up either.

If it's public, it's public. Don't put private stuff up in public.

It's not about that. Explicitly so. Did you read the thread you're responding to?

Yeah what this really needs is a bot block filter service where bots that do not conform to expectations can be bulk banned. It is too much effort to do this manually but a larger service can work.

Reply on news.ycombinator.com