Is it time for a web crawling code of conduct?

readwriteweb.com

4 pointsjdrock2 comments

Upstanding crawlers already follow this and more; rogue crawlers won't be influenced by such a code-of-conduct.

Furthermore, unless a crawler is trying really hard to disguise itself, site owners have the ultimate 'self-help' available to them: just block bad crawlers entirely.

Any formal code-of-conduct beyond what are already customary practices (like following robots.txt rules) thus strikes me as superfluous.

My hope in making a code of conduct accepted by legitimate web crawling companies is that it lets customers and users more easily decide which products to use.

It also helps in distancing legitimate companies from illicit ones.

Reply on news.ycombinator.com