Ask HN: What's a good general seed list for a web crawler
I'm developing a web crawler on top of a large distributed computer. As part of the testing process, I want to keep a background job running to keep crawling the web over and over. I was wondering if anyone had some ideas of a general seed list from which the crawler could reach a wide variety of links. It would be great if the links it traversed were a good representation of the Internet as a whole, taking into account content variety, frequency of updates, and other variables.