They're very useful in large scale web crawling/scraping. I use them for a number of things in this field.