Hacker News Reader

Comment by neilk | Hacker News Reader

Their robots.txt is password protected now. (?)

I'm not sure what a crawler is supposed to conclude in that case.

I don't see a lot of cpedia content on Google yet. The only links I can see are clearly due to the threads here and on sites like Reddit.

View on HN

Last I checked, Google's crawler treated 4xx errors on a /robots.txt (which includes unauthorized/not-allowed) the same as 404-not-found. (In most cases, it's a configuration error by the site owner, and for those sites where it's intentional, the operator is savvy enough to block crawler access to other URLs by password or user-agent if desired.)

Now it's this:

  User-agent: *
  Disallow: /imgsrv
  Disallow: /search
  Disallow: /pref
  Disallow: /suggest

> Their robots.txt is password protected now. (?)

It lets you in regardless of the username/password combination. I just logged in with this:

username: you morons

password: this defeats the whole damn purpose

I wonder if any of this gets logged at their end.

Reply on news.ycombinator.com