As an alternative to manual scraping, you can use CommonCrawl[0] or other open data sets, such as those provided by AWS[1]. That should alleviate any legal concerns (I think. I'm not a lawyer, but I'm sure CommonCrawl and Amazon have lawyers), and it's considerably faster than scraping. On top of that, you don't end up placing an unnecessary load on random websites.