Compliance with robots.txt is and always has been voluntary, and many crawlers have long ignored it, including Archive.org.