I've been developing the stack for The ContentMine (http://contentmine.org). In the next month or so, we will start scraping the entire scientific literature as it is published each day, and processing it through our 'fact extraction' pipeline.

Done so far:

- I've defined a JSON format for declarative web scrapers (ScraperJSON: https://github.com/ContentMine/scraperJSON)

- made a Node library for web scraping with ScraperJSON scrapers (thresher: https://github.com/ContentMine/thresher)

- as well as a command-line client (quickscrape: https://github.com/ContentMine/quickscrape)

- and a small library of ScraperJSON scrapers for scientific publishers that is about to start expanding rapidly (https://github.com/ContentMine/journal-scrapers).

Next step is to build the web interface that will let people compose data mining pipelines. Imagine something like:

- "give me a feed of all the articles in journals with 'Cancer' in the title that use HeLa cells in their methods"

- "alert me when a new paper comes out that mentions species X alongside a geographical reference"

- "find all the papers that mention my software in the methods but don't cite me"