Hacker News Reader

Comment by rodion_89 | Hacker News Reader

Their robots.txt file clearly asks to not be crawled at all.

  User-agent: *
  Disallow:

Incorrect. To configure your robots.txt to not be crawled at all use:

  User-agent: *
  Disallow: /

Allow indexing of everything with:

  User-agent: *
  Disallow:

It seems they are at this very moment struggling to change things.

You are totally right. Disregard everything I said.

Disallow nothing -> allow everything.