Using Scrapy to Build Your Own Dataset

Just in time. That XPath example is really useful for me.

I didn't finished the article yet, but I have one question. Is there ability in Scrapy to append new data after initial scrapping?

For example site at the moment of scrapping:

  Article 1  
  Article 2

Then we run scrapy again after some time and site will look like this:

  Article 1  
  Article 2  
  Article 3  
  Article 4

Can it just append article 3 and 4 to file/db table and stop scraping?

Scrapy is great. I have been using it at work to populate our solr index for our site search page. It is very extensible and powerful, but helps to understand Twisted if you need to do anything advanced.

Remember to scrape responsibly, check terms of service, robots.txt etc.

I used Scrapy extensively last term in my non-CS co-op. Really loved how quickly I could make beautiful dashboards with a bit of Python / VBA.

My problem with Scrapy was that there was a lot of manual work involved for each site, but perhaps that speaks more about my proficiency rather than Scrapy.

What's the next step from here in terms of gathering, cleaning, and analyzing data?

Scrapy? Really?

Scrapie is a transmissible spongiform ecephalopathy, similar to mad cow disease, in sheep and goats.

https://en.wikipedia.org/wiki/Scrapie