Hacker News Robots.txt Failure :(

2 pointsgmaster14406 comments

So here I am, sitting in front of my mac book pro wondering why my Hacker News app that I am developing began to crash. Turns out that Yahoo's YQL service began respecting robots.txt, and as a result is obeying HN's request to not allow scraping. With this, I beg you HN to allow at least minimal front page scraping for services like YQL. My app, that I hope you guys will be able to use in the near future, will be able to exist.

Thank you.

EXAMPLE: http://developer.yahoo.com/yql/console/?q=select%20%20from%20html%20where%20url%3D%22http://news.ycombinator.com%22%20and%20xpath%3D//table//table//td[%40class%3D%22title%22]/a#h=select%20%20from%20html%20where%20url%3D%22http%3A//news.ycombinator.com%22%20and%20xpath%3D%27//table//table//td%5B@class%3D%22title%22%5D/a%27

View on HN