Hacker News Reader

Remember to scrape responsibly, check terms of service, robots.txt etc.

Why? Isn’t it better to use a vps or two (or even just a vpn), and strong rate limiting? I.e. try not to be noticed regardless of the robots.txt.

Because it's considered polite.

A lot of robots.txt exist to help crawlers automatically know which URLs are safe and which ones are mutable (i.e. the sign up link).

You should only ignore the robots.txt if you are very careful about what you are doing and have a very good reason for doing so.