Hacker News Reader

PG talked about blocking all non-Google crawlers in an attempt to cut down on traffic ... not sure if he ever did or not

There doesn't seem to be anything about it in HN's robot.txt.

Of course, it may have been implemented in other ways.

What are those "x" URLs blocked in the robots.txt for HN?