What does this mean for http://www.searchyc.com? Will it stop too? (Afterall, it must be a bot too, and a robots.txt like this implies that no bots /should/ scrape the site.) Or will it be allowed?
That was my immediate question too. In addition to searchyc, we lose every other HN mash-up present and future.
Maybe this will finally bring about an HN API.
robots.txt is really only like a note on the front door of your unlocked house that says "please only look at these items in these rooms." or "please do not look in my house" If someone wants to write a program to index the site they can still do so. Google, MSN, Bing, yahoo, et. al. will respect the robots.txt file though.