Using Sitemaps in Web Scraping

Very useful for some big eshops if they list their sitemaps. However not all stores have sitemaps and even robots.txt site...

Good article, useful general tips that work for many scraping tools and platforms.