Recursive scraping:
1) can consume a lot of storage really fast
2) makes IA a bot rather than a user service
3) is already done on a case by case basis by ArchiveTeam allowing IA to stay away from the previous two problems.
1) can consume a lot of storage really fast
2) makes IA a bot rather than a user service
3) is already done on a case by case basis by ArchiveTeam allowing IA to stay away from the previous two problems.
I realize it's a hard problem, but I really wish there were a way to automate more of it. Some of these communities are too small for anyone to bother preserving the pages manually, and I don't imagine we'd even show up on ArchiveTeam's radar. But they're not large pages, and basically static. I don't think they'd be a huge burden to store and maintain. It seems like some sort of a coverage + size metric would be pretty effective at guiding an automated scan such that you'd be able to preserve things liked this without needing humans to go and manually archive each and every page.
AT just asks to be given a few weeks/months of notice depending on how many TBs (MBs?) the crawl needs to be and not to throttle/ban their clients.