Additionally, the interface to web crawlers (robots.txt) is well defined.