As a side-note, Internet Archive is starting to ignore robots.txt rules for this exact reason : https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...