Hacker News Reader

Comment by chaps | Hacker News Reader

I worked with y'all as a volunteer back when trump was pushing to drop global warming sites from gov websites. I also use the wayback machine professionally on a regular basis. The work y'all do is genuinely appreciated to say the least.

That said.. damn I really wish y'all would revisit some of your fundamentals like recursive scraping and making sure your scraping is whole and complete before working on filecoin and other needlessly flashy systems. I'm genuinely worried y'all are digging yourselves into a technical pit that you can't get out of and it will hurt or even kill your goals.

Cheers.

View on HN

Recursive scraping:

1) can consume a lot of storage really fast

2) makes IA a bot rather than a user service

3) is already done on a case by case basis by ArchiveTeam allowing IA to stay away from the previous two problems.

There's an old school website I wanted to access that's long gone. It's only partially preserved in the Wayback Machine. I'm glad it's there at all, but some of the specific information I was looking for was never captured, and now there's no way to recover it.

I realize it's a hard problem, but I really wish there were a way to automate more of it. Some of these communities are too small for anyone to bother preserving the pages manually, and I don't imagine we'd even show up on ArchiveTeam's radar. But they're not large pages, and basically static. I don't think they'd be a huge burden to store and maintain. It seems like some sort of a coverage + size metric would be pretty effective at guiding an automated scan such that you'd be able to preserve things liked this without needing humans to go and manually archive each and every page.

Pretty much. Your idea of getting coverage could probably work well to make sure IA doesn't "turn into a bot" (though it's practically the same, just without the permanent downloading).

https://wiki.archiveteam.org/index.php/ArchiveBot is available if you would like to help save things that are still up.

I never got turned down sending an IRC message "x is going down in a few days" to the AT channels. https://wiki.archiveteam.org/index.php/Archiveteam:IRC

AT just asks to be given a few weeks/months of notice depending on how many TBs (MBs?) the crawl needs to be and not to throttle/ban their clients.

What’s recursive scraping (presumably, following internal links), and what’s broken about it?

Yep, that's exactly it. Recursive scraping, in theory, would remove a lot gaps on IA. It's really hard to do generically and "right" though. Lots of one-off.. strangeness.. especially from older sites and sites that auto-generate URLs at render time.

Reply on news.ycombinator.com