Hacker News Reader

Hacker News

Comment by pearjuice | Hacker News Reader

Parent Full thread

pearjuice

SEO, click-through rate et cetera

View on HN

Hacker News isn't even listed on search engines because it disallows Googlebot.

It is definitely listed.

Hm, that's a recent change, then. HN wasn't listed just a week or two ago.

I've been using Google to find things on HN for a few years.

The file from January 2013 appears just as permissive:

http://web.archive.org/web/20130109093744/http://news.ycombi...

I have been too. I was surprised to find that HN didn't show up on Google a couple weeks ago. Yes, it looks like I was wrong. No, pg isn't changing titles because of SEO.

Maybe I'm missing something but

https://news.ycombinator.com/robots.txt

shows no such thing...

I dont' think that's true. https://news.ycombinator.com/robots.txt

  User-Agent: * 
  Disallow: /x?
  Disallow: /vote?
  Disallow: /reply?
  Disallow: /submitted?
  Disallow: /submitlink?
  Disallow: /threads?
  Crawl-delay: 30

Links in comments are marked nofollow but submissions are not.

Reply on news.ycombinator.com