Hacker News Reader

Comment by gcr | Hacker News Reader

I don't understand. If this virtual machine:

- Downloads dying web pages, and

- Uploads them to the Wayback Machine at the Internet Archive,

That means I'm not saving the Internet Archive any bandwidth at all, and this is no more efficient than them just downloading the site themselves.

The virtual machine itself is 174MB. This times however many volunteers means distributing the virtual machine is probably more stressful than the actual archiving operation.

View on HN

As mentioned, the issue is getting around YDOD throttling (and that the shutdown is happening in 10 days).

If you have proper boxen, you don't need the VM. Here's how to get it up and running on a clean Ubuntu setup for example:

  sudo apt-get install build-essential -y
  sudo apt-get install git -y
  sudo apt-get install libgnutls-dev -y
  sudo apt-get install liblua5.1-dev -y
  sudo apt-get install python-distribute -y
  sudo easy_install pip 
  sudo pip install seesaw
  git clone https://github.com/ArchiveTeam/yahoo-upcoming-grab.git
  cd yahoo-upcoming-grab
  ./get-wget-lua.sh 
  run-pipeline pipeline.py [YOURNAMEHERE]

Does that download posterous as well, or only upcoming?

For Posterous you'd need to check out https://github.com/ArchiveTeam/posterous-grab and run that particular script I believe.

Yahoo and others either rate limit or do temporary IP bans if you access too many pages too quickly. Distributing the tools through a VM distributes the workload in a pretty easy-to-setup way.

The Internet Archive crawler is polite, going at a slow rate.

With a deadline looming so soon, a more aggressive effort is called for.

It does actually save quite a lot of bandwidth in the process. The giles are gzipped as they are uploaded to the Archive Teams server, so your virtual machine will do a lot more download than it does upload.

Reply on news.ycombinator.com