robots.txt lacks nuance about copyright, it doesn't really serve the same purpose.