Archiveteam member here. Robots.txt is useful for a site to avoid having search-engine spiders to blindly wander down an infinite hallway.

Archiveteam projects are different because we run closely-monitored, highly targeted crawls. If a site has an infinite hallway in it, we'll notice that and exclude it while taking steps to replicate enough of it to retain its valuable attributes. We also do our best to avoid retrieving content more times than necessary. If you operate a database, we try to get enough pages to replicate a good portion of the underlying data.