Hacker News Reader

Comment by highace | Hacker News Reader

It's cloaked so only google can see it - try changing your user agent to match googlebot's.

Using the googlebot User-Agent string gives me:

  User-agent: *
  Disallow:

This robots.txt should allow all bots to search the entire website. However, I think Google also penalizes websites that serve different content to googlebot than to non-bot user agents.

This is the same as what I see, using a standard-issue Firefox UA string.

they updated it

That's weird because http://blog.digg.com/robots.txt does not cloak. Why cloak one and leave the other?

  Sitemap: http://blog.digg.com/sitemap-pages.xml
  Sitemap: http://blog.digg.com/sitemap1.xml

  User-agent: *
  Disallow: /private
  Disallow: /random
  Disallow: /day
  Crawl-delay: 1

Reply on news.ycombinator.com