Try out Nutch - http://lucene.apache.org/nutch/ or the one archive.org uses Heretix - http://crawler.archive.org/