Hacker News Reader

I don’t disagree with you about robots.txt… however, what _is_ the right tool for the job?

auth, If you don't want content to be publicly accessible, don't make it public.